Sep 07 10:09:30.493032 ip-10-0-128-191 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 07 10:09:30.493045 ip-10-0-128-191 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 07 10:09:30.493055 ip-10-0-128-191 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 07 10:09:30.493275 ip-10-0-128-191 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 07 10:09:40.742424 ip-10-0-128-191 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 07 10:09:40.742445 ip-10-0-128-191 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6e8d8521fe2946d1a2f42408000df646 --
Sep 07 10:11:31.944256 ip-10-0-128-191 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 10:11:32.262047 ip-10-0-128-191 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:11:32.262047 ip-10-0-128-191 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 10:11:32.262047 ip-10-0-128-191 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:11:32.262047 ip-10-0-128-191 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 10:11:32.262047 ip-10-0-128-191 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 10:11:32.263530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.263435    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 10:11:32.266315 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266294    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:32.266315 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266311    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:32.266315 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266317    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:32.266315 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266322    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266327    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266331    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266334    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266339    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266342    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266346    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266350    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266354    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266358    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266362    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266366    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266370    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266373    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266377    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266381    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266387    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266393    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266400    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266405    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:32.266543 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266409    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266414    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266418    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266422    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266426    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266430    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266437    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266443    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266449    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266454    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266458    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266463    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266467    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266471    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266476    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266480    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266485    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266489    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266493    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266497    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:32.267342 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266509    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266513    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266518    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266522    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266526    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266531    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266535    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266539    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266543    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266547    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266550    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266556    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266560    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266564    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266568    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266572    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266577    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266581    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266585    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:32.268139 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266589    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266593    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266600    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266607    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266613    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266618    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266623    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266630    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266634    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266639    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266643    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266648    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266652    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266656    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266661    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266665    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266669    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266674    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266678    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:32.268642 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266682    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266686    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266690    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266694    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266698    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266703    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266707    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266712    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266716    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266720    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266725    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266729    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266733    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266737    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266740    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266745    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266749    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.266754    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268703    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268714    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:32.269170 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268717    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268720    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268723    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268726    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268729    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268731    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268734    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268737    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268740    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268742    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268745    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268748    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268750    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268753    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268755    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268758    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268760    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268763    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268765    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268768    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:32.269661 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268771    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268774    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268777    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268779    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268782    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268784    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268787    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268789    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268792    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268795    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268798    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268801    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268804    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268806    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268809    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268811    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268814    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268817    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268819    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268821    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:32.270204 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268824    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268827    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268829    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268831    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268834    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268837    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268839    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268841    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268846    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268849    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268851    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268854    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268856    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268859    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268861    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268864    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268866    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268869    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268871    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268874    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:32.270692 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268876    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268879    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268882    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268887    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268891    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268895    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268897    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268900    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268903    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268906    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268909    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268911    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268914    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268916    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268921    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268924    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268927    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268929    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268932    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:32.271198 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268935    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268937    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268940    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268943    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268945    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268949    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268966    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268969    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268972    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268975    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268977    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268980    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268983    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268986    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268988    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268992    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268995    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.268998    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269410    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:32.271666 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269416    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269420    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269425    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269428    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269431    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269433    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269436    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269438    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269441    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269444    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269446    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269449    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269451    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269454    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269457    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269460    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269462    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269465    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269468    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:32.272145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269471    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269474    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269476    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269479    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269481    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269484    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269486    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269489    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269493    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269497    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269500    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269504    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269507    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269510    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269512    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269515    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269518    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269521    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269524    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:32.272610 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269526    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269529    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269533    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269535    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269538    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269540    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269543    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269546    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269548    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269551    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269560    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269563    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269566    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269568    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269571    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269574    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269576    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269579    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269581    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269584    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:32.273091 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269586    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269589    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269592    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269595    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269597    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269600    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269602    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269605    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269608    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269611    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269614    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269616    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269619    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269622    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269624    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269627    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269630    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269632    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269635    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269637    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:32.273585 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269640    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269643    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269645    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269648    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269651    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269654    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269656    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269659    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269661    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269665    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269667    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269670    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269672    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269677    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269679    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269682    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269685    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269687    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269690    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.269692    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:32.274121 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269762    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269794    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269801    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269806    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269810    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269814    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269818    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269822    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269825    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269828    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269831    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269834    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269837    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269840    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269844    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269847    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269850    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269853    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269857    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269860    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269863    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269866    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269869    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269873    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 10:11:32.274597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269875    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269883    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269886    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269889    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269892    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269895    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269898    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269901    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269905    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269907    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269910    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269913    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269916    2560 flags.go:64] FLAG: --enable-server="true"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269918    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269923    2560 flags.go:64] FLAG: --event-burst="100"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269926    2560 flags.go:64] FLAG: --event-qps="50"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269928    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269932    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269934    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269938    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269941    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269945    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269948    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269963    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269966    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 10:11:32.275248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269971    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269974    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269977    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269980    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269983    2560 flags.go:64] FLAG: --feature-gates=""
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269987    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269990    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269993    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.269997    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270000    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270003    2560 flags.go:64] FLAG: --help="false"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270006    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270009    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270012    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270015    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270018    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270021    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270024    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270027    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270030    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270032    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270035    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270039    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270042    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 07 10:11:32.275875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270045    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270048    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270051    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270053    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270058    2560 flags.go:64] FLAG: --lock-file=""
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270060    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270063    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270066    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270071    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270075    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270078    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270081    2560 flags.go:64] FLAG: --logging-format="text"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270083    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270087    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270089    2560 flags.go:64] FLAG: --manifest-url=""
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270092    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270097    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270100    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270104    2560 flags.go:64] FLAG: --max-pods="110"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270107    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270110    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270113    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270116    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270119    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270122    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 10:11:32.276474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270125    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270132    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270135    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270138    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270141    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270144    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270149    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270152    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270155    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270158    2560 flags.go:64] FLAG: --port="10250"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270161    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270164    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04590d22526bfef1f"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270168    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270171    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270173    2560 flags.go:64] FLAG: --register-node="true"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270176    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270180    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270183    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270186    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270189    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270193    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270196    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270199    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270203    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270206    2560 flags.go:64] FLAG: --runonce="false"
Sep 07 10:11:32.277124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270209    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270212    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270215    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270218    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270221    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270224    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270227    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270230    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270233    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270236    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270238    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270241    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270244    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270247    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270249    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270254    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270257    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270260    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270264    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270269    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270272    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270274    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270277    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270280    2560 flags.go:64] FLAG: --v="2"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270284    2560 flags.go:64] FLAG: --version="false"
Sep 07 10:11:32.277858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270288    2560 flags.go:64] FLAG: --vmodule=""
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270292    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.270295    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270414    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270417    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270421    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270424    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270427    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270429    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270432    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270434    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270437    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270440    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270442    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270445    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270447    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270450    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270453    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270455    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270458    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:32.278473 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270461    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270464    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270468    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270472    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270476    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270481    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270486    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270489    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270491    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270494    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270497    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270499    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270502    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270505    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270508    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270510    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270513    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270516    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270518    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270521    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:32.278970 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270523    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270526    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270528    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270531    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270534    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270536    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270539    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270541    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270544    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270547    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270549    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270552    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270555    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270557    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270560    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270562    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270565    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270568    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270571    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270574    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:32.279469 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270576    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270579    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270582    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270584    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270587    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270590    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270593    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270596    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270599    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270602    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270604    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270608    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270612    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270615    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270618    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270621    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270624    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270627    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270630    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:32.279999 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270632    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270636    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270639    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270643    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270645    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270648    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270650    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270653    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270656    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270658    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270661    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270665    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270668    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270671    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270673    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270675    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270678    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270681    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270683    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:32.280468 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270686    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270689    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270692    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.270695    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.271269    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.278387    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.278402    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.278998    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.279005    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279059    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279065    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279069    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279072    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279074    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279077    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:32.280930 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279080    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279083    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279085    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279088    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279091    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279094    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279096    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279099    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279102    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279105    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279107    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279110    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279113    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279116    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279119    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279122    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279124    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279127    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279131    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279133    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:32.281327 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279136    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279139    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279141    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279144    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279147    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279150    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279153    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279155    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279158    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279160    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279163    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279166    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279168    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279171    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279173    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279177    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279181    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279184    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279187    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:32.281813 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279190    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279192    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279195    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279198    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279201    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279203    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279206    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279208    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279211    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279213    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279216    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279219    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279222    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279225    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279227    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279230    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279234    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279236    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279239    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279243    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:32.282281 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279247    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279250    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279252    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279255    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279258    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279260    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279263    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279266    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279268    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279271    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279273    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279276    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279278    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279281    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279283    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279286    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279288    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279291    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279293    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279296    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:32.282761 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279298    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279301    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279303    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279306    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279310    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279314    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279317    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279320    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279323    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279326    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279329    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279331    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279334    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279337    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:32.283253 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.279342    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279447    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279451    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279454    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279457    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279461    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279463    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279466    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279469    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279472    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279475    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279478    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279481    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279484    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279487    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279489    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279492    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279494    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279497    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279500    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 10:11:32.283600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279502    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279505    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279507    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279510    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279514    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279518    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279521    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279524    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279527    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279530    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279533    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279536    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279539    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279542    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279544    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279547    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279550    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279552    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279555    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 10:11:32.284095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279557    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279560    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279563    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279565    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279567    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279570    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279573    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279575    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279578    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279581    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279583    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279586    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279588    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279591    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279593    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279596    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279599    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279601    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279604    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279607    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 10:11:32.284555 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279610    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279613    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279615    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279618    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279621    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279623    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279625    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279628    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279631    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279633    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279636    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279638    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279642    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279645    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279648    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279650    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279653    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279656    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279659    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 10:11:32.285077 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279662    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279664    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279668    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279671    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279675    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279677    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279680    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279683    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279685    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279687    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279690    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279693    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279696    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279699    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279701    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279704    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279706    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279709    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279711    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 10:11:32.285550 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279714    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279717    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:32.279719    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.279724    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.279828    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.281860    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.282814    2560 server.go:1019] "Starting client certificate rotation"
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.282914    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.283571    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 10:11:32.286029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.285059    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 10:11:32.287858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.287840    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 10:11:32.298180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.298158    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 07 10:11:32.303619 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.303599    2560 log.go:25] "Validated CRI v1 image API"
Sep 07 10:11:32.305416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.305400    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 10:11:32.309422 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.309401    2560 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 dace4e20-7703-493b-a47b-9e39b4317aa2:/dev/nvme0n1p4 f63799c1-5a45-48de-829a-8c1476394ccc:/dev/nvme0n1p3]
Sep 07 10:11:32.309494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.309421    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 10:11:32.314650 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.314632    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 10:11:32.315153 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.315043    2560 manager.go:216] Machine: {Timestamp:2026-09-07 10:11:32.31377714 +0000 UTC m=+0.273407948 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099372 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a96069c9ae0f0ce13b922f544755 SystemUUID:ec24a960-69c9-ae0f-0ce1-3b922f544755 BootID:6e8d8521-fe29-46d1-a2f4-2408000df646 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:34:de:53:6a:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:34:de:53:6a:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:ab:19:c5:dd:93 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 10:11:32.315153 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.315149    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 10:11:32.315280 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.315224    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 10:11:32.317916 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.317888    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 10:11:32.318052 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.317919    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 10:11:32.318518 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.318508    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 10:11:32.318548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.318521    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 10:11:32.318548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.318528    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 10:11:32.319114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.319104    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 10:11:32.320052 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.320041    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 10:11:32.323822 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.323810    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 10:11:32.323980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.323969    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 10:11:32.324013 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.323998    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 10:11:32.324013 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.324012    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 10:11:32.324069 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.324031    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 10:11:32.324069 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.324047    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 07 10:11:32.324069 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.324057    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 10:11:32.324478 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.324455    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 10:11:32.324532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.324485    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 10:11:32.327492 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.327476    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 10:11:32.330175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.330162    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 10:11:32.330232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.330192    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 10:11:32.331690 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331676    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 10:11:32.331741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331698    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 10:11:32.331741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331709    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 10:11:32.331741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331718    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 10:11:32.331741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331730    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 10:11:32.331741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331738    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 10:11:32.331872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331758    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 10:11:32.331872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331766    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 10:11:32.331872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331776    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 10:11:32.331872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331784    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 10:11:32.331872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331795    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 10:11:32.331872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.331803    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 10:11:32.332697 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.332686    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 10:11:32.332732 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.332699    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 10:11:32.334211 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.334178    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 10:11:32.334285 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.334191    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 10:11:32.336490 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.336478    2560 server.go:1268] "Started kubelet"
Sep 07 10:11:32.336607 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.336572    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 10:11:32.336922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.336780    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 10:11:32.337082 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.337063    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 10:11:32.337377 ip-10-0-128-191 systemd[1]: Started Kubernetes Kubelet.
Sep 07 10:11:32.338606 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.338585    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4pfq9"
Sep 07 10:11:32.340026 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.339492    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 10:11:32.340138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.340109    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 10:11:32.341326 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.341284    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 10:11:32.342094 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.342078    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 10:11:32.343506 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.343489    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4pfq9"
Sep 07 10:11:32.344339 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.343221    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-191.ec2.internal.18d302325d7f241b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-191.ec2.internal,UID:ip-10-0-128-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-191.ec2.internal,},FirstTimestamp:2026-09-07 10:11:32.336452635 +0000 UTC m=+0.296083443,LastTimestamp:2026-09-07 10:11:32.336452635 +0000 UTC m=+0.296083443,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-191.ec2.internal,}"
Sep 07 10:11:32.347054 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.347038    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 10:11:32.347703 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.347685    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 10:11:32.347836 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.347822    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:32.348595 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348569    2560 factory.go:153] Registering CRI-O factory
Sep 07 10:11:32.348595 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348596    2560 factory.go:223] Registration of the crio container factory successfully
Sep 07 10:11:32.348729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348638    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 10:11:32.348729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348644    2560 factory.go:55] Registering systemd factory
Sep 07 10:11:32.348729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348650    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 07 10:11:32.348729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348661    2560 factory.go:103] Registering Raw factory
Sep 07 10:11:32.348729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348670    2560 manager.go:1196] Started watching for new ooms in manager
Sep 07 10:11:32.348986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348808    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 10:11:32.348986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348809    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 10:11:32.348986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348835    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 10:11:32.348986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348943    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 10:11:32.348986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.348968    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 10:11:32.349228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.349003    2560 manager.go:349] Starting recovery of all containers
Sep 07 10:11:32.349464 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.349435    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 07 10:11:32.349592 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.349569    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 07 10:11:32.359336 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.359116    2560 manager.go:354] Recovery completed
Sep 07 10:11:32.363325 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.363308    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:32.365216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.365200    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:32.365320 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.365230    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:32.365320 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.365244    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:32.365833 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.365816    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:32.366339 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.365825    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 10:11:32.366425 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.366342    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 10:11:32.366425 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.366360    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 10:11:32.366592 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.366513    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:32.366592 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.366544    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:32.366592 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.366558    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:32.367694 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.367680    2560 policy_none.go:49] "None policy: Start"
Sep 07 10:11:32.367761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.367696    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 10:11:32.367761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.367707    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 10:11:32.368895 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.368879    2560 policy_none.go:47] "Start"
Sep 07 10:11:32.392727 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.392515    2560 manager.go:337] "Starting Device Plugin manager"
Sep 07 10:11:32.392727 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.392651    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 10:11:32.392727 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.392664    2560 server.go:87] "Starting device plugin registration server"
Sep 07 10:11:32.392890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.392878    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 10:11:32.392934 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.392889    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 10:11:32.393050 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.393031    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 10:11:32.393129 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.393109    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 10:11:32.393129 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.393122    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 10:11:32.393650 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.393630    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 10:11:32.393738 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.393682    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:32.476641 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.476610    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 10:11:32.477813 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.477741    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 10:11:32.477813 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.477792    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 10:11:32.477928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.477816    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 10:11:32.477928 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.477859    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 10:11:32.480025 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.480009    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:32.493928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.493908    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:32.494643 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.494628    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:32.494741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.494652    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:32.494741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.494662    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:32.494741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.494680    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.502382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.502368    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.502445 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.502387    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-191.ec2.internal\": node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:32.511749 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.511730    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:32.578588 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.578566    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal"]
Sep 07 10:11:32.578672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.578630    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:32.579344 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.579329    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:32.579434 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.579357    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:32.579434 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.579372    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:32.580529 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.580515    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:32.580647 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.580632    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.580696 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.580661    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:32.581214 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.581198    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:32.581214 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.581207    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:32.581317 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.581228    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:32.581317 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.581229    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:32.581317 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.581244    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:32.581317 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.581252    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:32.582292 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.582277    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.582340 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.582303    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 10:11:32.582991 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.582975    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 10:11:32.583059 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.583001    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 10:11:32.583059 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.583010    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeHasSufficientPID"
Sep 07 10:11:32.598999 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.598977    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-191.ec2.internal\" not found" node="ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.601710 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.601694    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-191.ec2.internal\" not found" node="ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.612519 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.612493    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:32.713339 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.713318    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:32.750211 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.750190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6acb16e1cdb3c44a9ba2958443fb258-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"c6acb16e1cdb3c44a9ba2958443fb258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.750298 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.750215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6acb16e1cdb3c44a9ba2958443fb258-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"c6acb16e1cdb3c44a9ba2958443fb258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.750298 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.750233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c349833f5e025366a333ee7959fbf2b3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-191.ec2.internal\" (UID: \"c349833f5e025366a333ee7959fbf2b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.813421 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.813378    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:32.850862 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.850837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c349833f5e025366a333ee7959fbf2b3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-191.ec2.internal\" (UID: \"c349833f5e025366a333ee7959fbf2b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.850936 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.850873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6acb16e1cdb3c44a9ba2958443fb258-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"c6acb16e1cdb3c44a9ba2958443fb258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.850936 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.850899    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6acb16e1cdb3c44a9ba2958443fb258-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"c6acb16e1cdb3c44a9ba2958443fb258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.851049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.850972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6acb16e1cdb3c44a9ba2958443fb258-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"c6acb16e1cdb3c44a9ba2958443fb258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.851049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.850987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c349833f5e025366a333ee7959fbf2b3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-191.ec2.internal\" (UID: \"c349833f5e025366a333ee7959fbf2b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.851049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.851024    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6acb16e1cdb3c44a9ba2958443fb258-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal\" (UID: \"c6acb16e1cdb3c44a9ba2958443fb258\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.900977 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.900948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.905656 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:32.905639    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:32.914166 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:32.914146    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:33.014676 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:33.014654    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:33.115223 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:33.115167    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-191.ec2.internal\" not found"
Sep 07 10:11:34.599989 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.599799    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:34.659801 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.659780    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:34.671016 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.670996    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 10:11:34.672398 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.672385    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal"
Sep 07 10:11:34.679169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.679151    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 10:11:34.694316 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.694292    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 10:11:34.694429 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.694404    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 10:11:34.694504 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.694441    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 10:11:34.735541 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.735521    2560 apiserver.go:52] "Watching apiserver"
Sep 07 10:11:34.750492 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.750473    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 10:11:34.750866 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.750848    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-dbjml","openshift-network-diagnostics/network-check-target-zzfpd","openshift-network-operator/iptables-alerter-hjfhg","kube-system/global-pull-secret-syncer-jr9hg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx","openshift-cluster-node-tuning-operator/tuned-w9f25","openshift-dns/node-resolver-cjhxw","openshift-image-registry/node-ca-56glg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal","openshift-multus/multus-additional-cni-plugins-shht6","openshift-ovn-kubernetes/ovnkube-node-k4r66","kube-system/konnectivity-agent-sxqmk","kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal","openshift-multus/multus-ngwlm"]
Sep 07 10:11:34.752405 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.752387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.753524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.753506    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.754458 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.754422    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 10:11:34.754647 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.754631    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.754737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.754723    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 10:11:34.754808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.754761    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 10:11:34.754868 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.754811    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 10:11:34.754945 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.754931    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-68tkj\""
Sep 07 10:11:34.754945 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.754940    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 10:11:34.757398 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.757373    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 10:11:34.757398 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.757400    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 10:11:34.757557 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.757470    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:11:34.757624 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.757583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.757693 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.757650    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 10:11:34.757758 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.757742    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rw82\""
Sep 07 10:11:34.757843 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.757826    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-48zjp\""
Sep 07 10:11:34.757906 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.757850    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 10:11:34.758272 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.758259    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 10:11:34.758784 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.758768    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:34.758860 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.758831    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:34.759522 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.759507    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 10:11:34.759608 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.759537    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8wpgm\""
Sep 07 10:11:34.759877 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.759859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.759996 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.759937    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 10:06:33 +0000 UTC" deadline="2028-05-11 10:17:15.437098729 +0000 UTC"
Sep 07 10:11:34.760093 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.759997    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14688h5m40.677104993s"
Sep 07 10:11:34.760917 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.760903    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.761800 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.761786    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:11:34.761909 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.761878    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 10:11:34.762020 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.761956    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hzrdz\""
Sep 07 10:11:34.762138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.762124    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:34.762205 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.762190    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:34.762983 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.762966    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 10:11:34.763056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.762988    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 10:11:34.763056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763032    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l7t4m\""
Sep 07 10:11:34.763278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763265    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 10:11:34.763432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.763777 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnw7q\" (UniqueName: \"kubernetes.io/projected/679b04d9-df87-45a9-8bc1-8cb7d5c88758-kube-api-access-pnw7q\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.763850 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-modprobe-d\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.763850 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-run\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.763850 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-var-lib-kubelet\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.763959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc2nb\" (UniqueName: \"kubernetes.io/projected/1ce3557a-cc31-4782-9785-ed93485f273a-kube-api-access-nc2nb\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.763959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-system-cni-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.763959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-netns\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.763959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-multus-certs\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.763959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.763985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-k8s-cni-cncf-io\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tccdf\" (UniqueName: \"kubernetes.io/projected/ffd270e7-a91f-41cd-918e-c2650aa8f280-kube-api-access-tccdf\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-lib-modules\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-tuned\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-tmp\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764143    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-system-cni-dir\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-cni-binary-copy\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-socket-dir-parent\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-sys-fs\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.764234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764221    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-kubernetes\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-cni-multus\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-registration-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffd270e7-a91f-41cd-918e-c2650aa8f280-iptables-alerter-script\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-sys\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-cni-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764355    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-kubelet\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-hostroot\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-etc-kubernetes\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764421    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrds5\" (UniqueName: \"kubernetes.io/projected/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-kube-api-access-qrds5\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-device-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysconfig\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysctl-d\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-systemd\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.764566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-cni-binary-copy\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd270e7-a91f-41cd-918e-c2650aa8f280-host-slash\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-cnibin\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-cni-bin\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-conf-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-daemon-config\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764755    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-socket-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds2hx\" (UniqueName: \"kubernetes.io/projected/0e67e468-8cf2-4095-8878-e5698eaa805a-kube-api-access-ds2hx\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/679b04d9-df87-45a9-8bc1-8cb7d5c88758-hosts-file\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysctl-conf\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-host\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj9dr\" (UniqueName: \"kubernetes.io/projected/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-kube-api-access-zj9dr\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764957    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.764993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-cnibin\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.765013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.765172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.765028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/679b04d9-df87-45a9-8bc1-8cb7d5c88758-tmp-dir\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.765867 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.765548    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 10:11:34.765867 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.765548    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nw5cn\""
Sep 07 10:11:34.765867 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.765874    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 10:11:34.766051 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.766034    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 10:11:34.766160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.766058    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.766160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.766043    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 10:11:34.766160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.766115    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 10:11:34.766377 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.766360    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 10:11:34.767159 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.767146    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 10:11:34.767479 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.767461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.767479 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.767474    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nhwjb\""
Sep 07 10:11:34.767592 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.767514    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:34.768186 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.768157    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 10:11:34.768281 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.768262    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 10:11:34.768428 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.768416    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 10:11:34.768510 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.768428    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-csphj\""
Sep 07 10:11:34.768934 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.768918    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 10:11:34.771616 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.771599    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 10:11:34.794261 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.794241    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jnd84"
Sep 07 10:11:34.802328 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.802314    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jnd84"
Sep 07 10:11:34.860471 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.860455    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 10:11:34.865152 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865133    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfd233c5-b9bd-47b1-aee1-1013a76983b5-kubelet-config\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.865204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-etc-kubernetes\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrds5\" (UniqueName: \"kubernetes.io/projected/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-kube-api-access-qrds5\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysconfig\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.865251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-cni-binary-copy\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-etc-kubernetes\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-slash\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-cnibin\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysconfig\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-conf-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865370    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-conf-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-daemon-config\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865366    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-cnibin\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-socket-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.865439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865437    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnw7q\" (UniqueName: \"kubernetes.io/projected/679b04d9-df87-45a9-8bc1-8cb7d5c88758-kube-api-access-pnw7q\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysctl-conf\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865479    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865503    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/969e1cf8-0598-41cd-a6d5-626d5fc54ea7-agent-certs\") pod \"konnectivity-agent-sxqmk\" (UID: \"969e1cf8-0598-41cd-a6d5-626d5fc54ea7\") " pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865525    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-modprobe-d\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-netns\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-multus-certs\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b57956ef-2740-4da5-b4ce-78e476de56e9-ovn-node-metrics-cert\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlsqj\" (UniqueName: \"kubernetes.io/projected/232a763f-6ba8-4484-855b-ead84890b628-kube-api-access-vlsqj\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865649    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-modprobe-d\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysctl-conf\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-lib-modules\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-socket-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-netns\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-multus-certs\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.865928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-lib-modules\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-tmp\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-cni-binary-copy\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-socket-dir-parent\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-cni-binary-copy\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-sys-fs\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-socket-dir-parent\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-cni-bin\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-sys-fs\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-env-overrides\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.865989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-host\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-kubernetes\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-registration-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866085    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffd270e7-a91f-41cd-918e-c2650aa8f280-iptables-alerter-script\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.866672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-kubernetes\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866118    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-run-netns\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-registration-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-etc-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866178    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd270e7-a91f-41cd-918e-c2650aa8f280-host-slash\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd270e7-a91f-41cd-918e-c2650aa8f280-host-slash\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-sys\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-cni-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-kubelet\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-hostroot\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-cni-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-device-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866338    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-sys\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-systemd-units\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-multus-daemon-config\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysctl-d\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.867232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866443    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-kubelet\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-systemd\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866512    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-systemd\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-cni-binary-copy\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-ovn\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-hostroot\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e67e468-8cf2-4095-8878-e5698eaa805a-device-dir\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-sysctl-d\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866575    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-cni-netd\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffd270e7-a91f-41cd-918e-c2650aa8f280-iptables-alerter-script\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866611    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-cni-bin\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds2hx\" (UniqueName: \"kubernetes.io/projected/0e67e468-8cf2-4095-8878-e5698eaa805a-kube-api-access-ds2hx\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866690    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-cni-bin\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.867808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/679b04d9-df87-45a9-8bc1-8cb7d5c88758-hosts-file\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfd233c5-b9bd-47b1-aee1-1013a76983b5-dbus\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/679b04d9-df87-45a9-8bc1-8cb7d5c88758-hosts-file\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866741    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnk48\" (UniqueName: \"kubernetes.io/projected/b57956ef-2740-4da5-b4ce-78e476de56e9-kube-api-access-jnk48\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-host\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj9dr\" (UniqueName: \"kubernetes.io/projected/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-kube-api-access-zj9dr\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866853    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-host\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866877    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-cnibin\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/679b04d9-df87-45a9-8bc1-8cb7d5c88758-tmp-dir\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-var-lib-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-node-log\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dbh\" (UniqueName: \"kubernetes.io/projected/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-kube-api-access-n8dbh\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-cnibin\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.866981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-run\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-var-lib-kubelet\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.868322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867023    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc2nb\" (UniqueName: \"kubernetes.io/projected/1ce3557a-cc31-4782-9785-ed93485f273a-kube-api-access-nc2nb\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867027    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-run\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-system-cni-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-systemd\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-var-lib-kubelet\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867110    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-ovnkube-config\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/969e1cf8-0598-41cd-a6d5-626d5fc54ea7-konnectivity-ca\") pod \"konnectivity-agent-sxqmk\" (UID: \"969e1cf8-0598-41cd-a6d5-626d5fc54ea7\") " pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867148    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-system-cni-dir\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-serviceca\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ce3557a-cc31-4782-9785-ed93485f273a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-k8s-cni-cncf-io\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tccdf\" (UniqueName: \"kubernetes.io/projected/ffd270e7-a91f-41cd-918e-c2650aa8f280-kube-api-access-tccdf\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-run-k8s-cni-cncf-io\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867359    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-log-socket\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-tuned\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867486    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-system-cni-dir\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-kubelet\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.869421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-cni-multus\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.869421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.869421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce3557a-cc31-4782-9785-ed93485f273a-system-cni-dir\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.869421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-ovnkube-script-lib\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.869421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867596    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-host-var-lib-cni-multus\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.869421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.867622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/679b04d9-df87-45a9-8bc1-8cb7d5c88758-tmp-dir\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.869421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.869309    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-etc-tuned\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.869620 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.869445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-tmp\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.874095 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.874061    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:34.874192 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.874101    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:34.874192 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.874115    2560 projected.go:196] Error preparing data for projected volume kube-api-access-s8szd for pod openshift-network-diagnostics/network-check-target-zzfpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:34.874192 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.874176    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd podName:98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d nodeName:}" failed. No retries permitted until 2026-09-07 10:11:35.374154798 +0000 UTC m=+1.922669787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s8szd" (UniqueName: "kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd") pod "network-check-target-zzfpd" (UID: "98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:34.876722 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.876698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnw7q\" (UniqueName: \"kubernetes.io/projected/679b04d9-df87-45a9-8bc1-8cb7d5c88758-kube-api-access-pnw7q\") pod \"node-resolver-cjhxw\" (UID: \"679b04d9-df87-45a9-8bc1-8cb7d5c88758\") " pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:34.876837 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.876706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tccdf\" (UniqueName: \"kubernetes.io/projected/ffd270e7-a91f-41cd-918e-c2650aa8f280-kube-api-access-tccdf\") pod \"iptables-alerter-hjfhg\" (UID: \"ffd270e7-a91f-41cd-918e-c2650aa8f280\") " pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:34.877082 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.877048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrds5\" (UniqueName: \"kubernetes.io/projected/c50d79e9-af4e-4927-b4f4-2cfd83f3290e-kube-api-access-qrds5\") pod \"multus-ngwlm\" (UID: \"c50d79e9-af4e-4927-b4f4-2cfd83f3290e\") " pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:34.877885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.877864    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds2hx\" (UniqueName: \"kubernetes.io/projected/0e67e468-8cf2-4095-8878-e5698eaa805a-kube-api-access-ds2hx\") pod \"aws-ebs-csi-driver-node-hmqfx\" (UID: \"0e67e468-8cf2-4095-8878-e5698eaa805a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:34.878332 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.878311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc2nb\" (UniqueName: \"kubernetes.io/projected/1ce3557a-cc31-4782-9785-ed93485f273a-kube-api-access-nc2nb\") pod \"multus-additional-cni-plugins-shht6\" (UID: \"1ce3557a-cc31-4782-9785-ed93485f273a\") " pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:34.878807 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.878784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj9dr\" (UniqueName: \"kubernetes.io/projected/5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46-kube-api-access-zj9dr\") pod \"tuned-w9f25\" (UID: \"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46\") " pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:34.968203 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-cni-bin\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968203 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-env-overrides\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-host\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-run-netns\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-etc-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-cni-bin\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-host\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968303    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-etc-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-run-netns\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-systemd-units\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968341    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-systemd-units\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968368    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-ovn\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-cni-netd\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-ovn\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-cni-netd\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfd233c5-b9bd-47b1-aee1-1013a76983b5-dbus\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnk48\" (UniqueName: \"kubernetes.io/projected/b57956ef-2740-4da5-b4ce-78e476de56e9-kube-api-access-jnk48\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bfd233c5-b9bd-47b1-aee1-1013a76983b5-dbus\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-env-overrides\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-var-lib-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968670    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-node-log\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-var-lib-openvswitch\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968710    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-node-log\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.968821 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.968724    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.968771    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs podName:232a763f-6ba8-4484-855b-ead84890b628 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:35.468757195 +0000 UTC m=+2.017272164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs") pod "network-metrics-daemon-dbjml" (UID: "232a763f-6ba8-4484-855b-ead84890b628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dbh\" (UniqueName: \"kubernetes.io/projected/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-kube-api-access-n8dbh\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-systemd\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-ovnkube-config\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/969e1cf8-0598-41cd-a6d5-626d5fc54ea7-konnectivity-ca\") pod \"konnectivity-agent-sxqmk\" (UID: \"969e1cf8-0598-41cd-a6d5-626d5fc54ea7\") " pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-serviceca\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-run-systemd\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968876    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-log-socket\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968910    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-log-socket\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-kubelet\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.968992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-ovnkube-script-lib\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfd233c5-b9bd-47b1-aee1-1013a76983b5-kubelet-config\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-kubelet\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.969065    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-slash\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.969554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/969e1cf8-0598-41cd-a6d5-626d5fc54ea7-agent-certs\") pod \"konnectivity-agent-sxqmk\" (UID: \"969e1cf8-0598-41cd-a6d5-626d5fc54ea7\") " pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:34.969133    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret podName:bfd233c5-b9bd-47b1-aee1-1013a76983b5 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:35.469118284 +0000 UTC m=+2.017633265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret") pod "global-pull-secret-syncer-jr9hg" (UID: "bfd233c5-b9bd-47b1-aee1-1013a76983b5") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bfd233c5-b9bd-47b1-aee1-1013a76983b5-kubelet-config\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b57956ef-2740-4da5-b4ce-78e476de56e9-ovn-node-metrics-cert\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b57956ef-2740-4da5-b4ce-78e476de56e9-host-slash\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlsqj\" (UniqueName: \"kubernetes.io/projected/232a763f-6ba8-4484-855b-ead84890b628-kube-api-access-vlsqj\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-ovnkube-config\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-serviceca\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969492    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/969e1cf8-0598-41cd-a6d5-626d5fc54ea7-konnectivity-ca\") pod \"konnectivity-agent-sxqmk\" (UID: \"969e1cf8-0598-41cd-a6d5-626d5fc54ea7\") " pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:11:34.970106 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.969931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b57956ef-2740-4da5-b4ce-78e476de56e9-ovnkube-script-lib\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.971235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.971215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b57956ef-2740-4da5-b4ce-78e476de56e9-ovn-node-metrics-cert\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.971349 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.971272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/969e1cf8-0598-41cd-a6d5-626d5fc54ea7-agent-certs\") pod \"konnectivity-agent-sxqmk\" (UID: \"969e1cf8-0598-41cd-a6d5-626d5fc54ea7\") " pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:11:34.978595 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.978566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlsqj\" (UniqueName: \"kubernetes.io/projected/232a763f-6ba8-4484-855b-ead84890b628-kube-api-access-vlsqj\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:34.978776 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.978755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dbh\" (UniqueName: \"kubernetes.io/projected/eb2119b6-8f00-4d3d-a2f8-3689940c55fd-kube-api-access-n8dbh\") pod \"node-ca-56glg\" (UID: \"eb2119b6-8f00-4d3d-a2f8-3689940c55fd\") " pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:34.978817 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.978793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnk48\" (UniqueName: \"kubernetes.io/projected/b57956ef-2740-4da5-b4ce-78e476de56e9-kube-api-access-jnk48\") pod \"ovnkube-node-k4r66\" (UID: \"b57956ef-2740-4da5-b4ce-78e476de56e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:34.992269 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:34.992246    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc349833f5e025366a333ee7959fbf2b3.slice/crio-154e213e22adbe000b05728548a29a0f675ff56614772bfa8e50a0883f7e5a5b WatchSource:0}: Error finding container 154e213e22adbe000b05728548a29a0f675ff56614772bfa8e50a0883f7e5a5b: Status 404 returned error can't find the container with id 154e213e22adbe000b05728548a29a0f675ff56614772bfa8e50a0883f7e5a5b
Sep 07 10:11:34.992448 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:34.992427    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6acb16e1cdb3c44a9ba2958443fb258.slice/crio-9dc7129306b1c95116d11e82f0ad1a496c7ac795f305f7b0431b2f8ec091459f WatchSource:0}: Error finding container 9dc7129306b1c95116d11e82f0ad1a496c7ac795f305f7b0431b2f8ec091459f: Status 404 returned error can't find the container with id 9dc7129306b1c95116d11e82f0ad1a496c7ac795f305f7b0431b2f8ec091459f
Sep 07 10:11:34.996847 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:34.996830    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:11:35.014169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.014152    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:35.093266 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.093221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-shht6"
Sep 07 10:11:35.099231 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.099202    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cjhxw"
Sep 07 10:11:35.102236 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.102210    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce3557a_cc31_4782_9785_ed93485f273a.slice/crio-bc70bf408faa6d79efd2748cd7c082b3dfae8acc7e2bb76758a9a2b44e7c5c52 WatchSource:0}: Error finding container bc70bf408faa6d79efd2748cd7c082b3dfae8acc7e2bb76758a9a2b44e7c5c52: Status 404 returned error can't find the container with id bc70bf408faa6d79efd2748cd7c082b3dfae8acc7e2bb76758a9a2b44e7c5c52
Sep 07 10:11:35.107602 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.107584    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679b04d9_df87_45a9_8bc1_8cb7d5c88758.slice/crio-264399d7d7fe98e1db1424d19d5c3b0cd03e8de21e2310b9f6cc91ed75637297 WatchSource:0}: Error finding container 264399d7d7fe98e1db1424d19d5c3b0cd03e8de21e2310b9f6cc91ed75637297: Status 404 returned error can't find the container with id 264399d7d7fe98e1db1424d19d5c3b0cd03e8de21e2310b9f6cc91ed75637297
Sep 07 10:11:35.117369 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.117353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hjfhg"
Sep 07 10:11:35.126256 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.126238    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd270e7_a91f_41cd_918e_c2650aa8f280.slice/crio-b9363ec3564364d0703ddc532b11856ccc2f6a7818d9014d75bfdf51b7f4bfb4 WatchSource:0}: Error finding container b9363ec3564364d0703ddc532b11856ccc2f6a7818d9014d75bfdf51b7f4bfb4: Status 404 returned error can't find the container with id b9363ec3564364d0703ddc532b11856ccc2f6a7818d9014d75bfdf51b7f4bfb4
Sep 07 10:11:35.136541 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.136525    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ngwlm"
Sep 07 10:11:35.144791 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.144770    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50d79e9_af4e_4927_b4f4_2cfd83f3290e.slice/crio-12c2c2b0559f16f1f59765497df565b1755397cd3527e673877f2737031012fd WatchSource:0}: Error finding container 12c2c2b0559f16f1f59765497df565b1755397cd3527e673877f2737031012fd: Status 404 returned error can't find the container with id 12c2c2b0559f16f1f59765497df565b1755397cd3527e673877f2737031012fd
Sep 07 10:11:35.151599 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.151584    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9f25"
Sep 07 10:11:35.158312 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.158288    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2505c0_2b3f_41c8_a8e8_ebf813ba8d46.slice/crio-2481acbeeeea62bdc7c757ef8781f289d07abad3b28be4cc85eea2859bd96301 WatchSource:0}: Error finding container 2481acbeeeea62bdc7c757ef8781f289d07abad3b28be4cc85eea2859bd96301: Status 404 returned error can't find the container with id 2481acbeeeea62bdc7c757ef8781f289d07abad3b28be4cc85eea2859bd96301
Sep 07 10:11:35.170467 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.170449    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx"
Sep 07 10:11:35.177731 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.177715    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e67e468_8cf2_4095_8878_e5698eaa805a.slice/crio-e3a697a9084745bcc0013d42017ee62079045c9a29f4feefd7afb48e3cd02e88 WatchSource:0}: Error finding container e3a697a9084745bcc0013d42017ee62079045c9a29f4feefd7afb48e3cd02e88: Status 404 returned error can't find the container with id e3a697a9084745bcc0013d42017ee62079045c9a29f4feefd7afb48e3cd02e88
Sep 07 10:11:35.188199 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.188174    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:11:35.193981 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.193964    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:11:35.196095 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.196063    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57956ef_2740_4da5_b4ce_78e476de56e9.slice/crio-1557aaff1fb9b7fcd254f2e3bfedba598c01cbbb5ac3871d230e068c2f03a71a WatchSource:0}: Error finding container 1557aaff1fb9b7fcd254f2e3bfedba598c01cbbb5ac3871d230e068c2f03a71a: Status 404 returned error can't find the container with id 1557aaff1fb9b7fcd254f2e3bfedba598c01cbbb5ac3871d230e068c2f03a71a
Sep 07 10:11:35.201583 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.201566    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969e1cf8_0598_41cd_a6d5_626d5fc54ea7.slice/crio-538ae729b6148adcd096e1e677daa67f43c5b651ca94b822ebcdeb55f4a69731 WatchSource:0}: Error finding container 538ae729b6148adcd096e1e677daa67f43c5b651ca94b822ebcdeb55f4a69731: Status 404 returned error can't find the container with id 538ae729b6148adcd096e1e677daa67f43c5b651ca94b822ebcdeb55f4a69731
Sep 07 10:11:35.208512 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.208499    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-56glg"
Sep 07 10:11:35.216451 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:11:35.216435    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2119b6_8f00_4d3d_a2f8_3689940c55fd.slice/crio-103f47a7bd72ebf7680deaab983596150ba3099affd5f2672aebe046148e6780 WatchSource:0}: Error finding container 103f47a7bd72ebf7680deaab983596150ba3099affd5f2672aebe046148e6780: Status 404 returned error can't find the container with id 103f47a7bd72ebf7680deaab983596150ba3099affd5f2672aebe046148e6780
Sep 07 10:11:35.343495 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.343439    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:35.472706 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.472674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:35.472850 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.472727    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:35.472850 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.472789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:35.472967 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.472941    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:35.472967 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.472963    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:35.473063 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.472975    2560 projected.go:196] Error preparing data for projected volume kube-api-access-s8szd for pod openshift-network-diagnostics/network-check-target-zzfpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:35.473063 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.473029    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd podName:98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d nodeName:}" failed. No retries permitted until 2026-09-07 10:11:36.47301044 +0000 UTC m=+3.021525422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8szd" (UniqueName: "kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd") pod "network-check-target-zzfpd" (UID: "98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:35.473438 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.473419    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:35.473518 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.473475    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret podName:bfd233c5-b9bd-47b1-aee1-1013a76983b5 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:36.473460448 +0000 UTC m=+3.021975417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret") pod "global-pull-secret-syncer-jr9hg" (UID: "bfd233c5-b9bd-47b1-aee1-1013a76983b5") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:35.473580 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.473536    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:35.473580 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.473568    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs podName:232a763f-6ba8-4484-855b-ead84890b628 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:36.473557931 +0000 UTC m=+3.022072901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs") pod "network-metrics-daemon-dbjml" (UID: "232a763f-6ba8-4484-855b-ead84890b628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:35.805556 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.805344    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:06:34 +0000 UTC" deadline="2028-04-04 01:34:52.33779937 +0000 UTC"
Sep 07 10:11:35.805556 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.805379    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13791h23m16.532425552s"
Sep 07 10:11:35.891651 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.891178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:35.891651 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.891322    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:35.891651 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.891366    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:35.891651 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.891463    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:35.891651 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.891617    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:35.892004 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:35.891712    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:35.916717 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.916631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" event={"ID":"c6acb16e1cdb3c44a9ba2958443fb258","Type":"ContainerStarted","Data":"9dc7129306b1c95116d11e82f0ad1a496c7ac795f305f7b0431b2f8ec091459f"}
Sep 07 10:11:35.953124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.953094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-56glg" event={"ID":"eb2119b6-8f00-4d3d-a2f8-3689940c55fd","Type":"ContainerStarted","Data":"103f47a7bd72ebf7680deaab983596150ba3099affd5f2672aebe046148e6780"}
Sep 07 10:11:35.973773 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.973745    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ngwlm" event={"ID":"c50d79e9-af4e-4927-b4f4-2cfd83f3290e","Type":"ContainerStarted","Data":"12c2c2b0559f16f1f59765497df565b1755397cd3527e673877f2737031012fd"}
Sep 07 10:11:35.989997 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:35.989962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hjfhg" event={"ID":"ffd270e7-a91f-41cd-918e-c2650aa8f280","Type":"ContainerStarted","Data":"b9363ec3564364d0703ddc532b11856ccc2f6a7818d9014d75bfdf51b7f4bfb4"}
Sep 07 10:11:36.007665 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.007601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjhxw" event={"ID":"679b04d9-df87-45a9-8bc1-8cb7d5c88758","Type":"ContainerStarted","Data":"264399d7d7fe98e1db1424d19d5c3b0cd03e8de21e2310b9f6cc91ed75637297"}
Sep 07 10:11:36.022376 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.022345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shht6" event={"ID":"1ce3557a-cc31-4782-9785-ed93485f273a","Type":"ContainerStarted","Data":"bc70bf408faa6d79efd2748cd7c082b3dfae8acc7e2bb76758a9a2b44e7c5c52"}
Sep 07 10:11:36.029262 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.029237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxqmk" event={"ID":"969e1cf8-0598-41cd-a6d5-626d5fc54ea7","Type":"ContainerStarted","Data":"538ae729b6148adcd096e1e677daa67f43c5b651ca94b822ebcdeb55f4a69731"}
Sep 07 10:11:36.033400 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.033366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"1557aaff1fb9b7fcd254f2e3bfedba598c01cbbb5ac3871d230e068c2f03a71a"}
Sep 07 10:11:36.050343 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.049966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx" event={"ID":"0e67e468-8cf2-4095-8878-e5698eaa805a","Type":"ContainerStarted","Data":"e3a697a9084745bcc0013d42017ee62079045c9a29f4feefd7afb48e3cd02e88"}
Sep 07 10:11:36.072387 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.072233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9f25" event={"ID":"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46","Type":"ContainerStarted","Data":"2481acbeeeea62bdc7c757ef8781f289d07abad3b28be4cc85eea2859bd96301"}
Sep 07 10:11:36.077400 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.077375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" event={"ID":"c349833f5e025366a333ee7959fbf2b3","Type":"ContainerStarted","Data":"154e213e22adbe000b05728548a29a0f675ff56614772bfa8e50a0883f7e5a5b"}
Sep 07 10:11:36.176634 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.176608    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:36.492920 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.492837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:36.492920 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.492893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:36.493174 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.492927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:36.493174 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:36.493046    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:36.493174 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:36.493119    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret podName:bfd233c5-b9bd-47b1-aee1-1013a76983b5 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:38.493101181 +0000 UTC m=+5.041616152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret") pod "global-pull-secret-syncer-jr9hg" (UID: "bfd233c5-b9bd-47b1-aee1-1013a76983b5") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:36.493609 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:36.493586    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:36.493692 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:36.493643    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs podName:232a763f-6ba8-4484-855b-ead84890b628 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:38.493627884 +0000 UTC m=+5.042142853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs") pod "network-metrics-daemon-dbjml" (UID: "232a763f-6ba8-4484-855b-ead84890b628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:36.493756 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:36.493718    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:36.493756 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:36.493734    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:36.493756 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:36.493747    2560 projected.go:196] Error preparing data for projected volume kube-api-access-s8szd for pod openshift-network-diagnostics/network-check-target-zzfpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:36.493901 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:36.493781    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd podName:98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d nodeName:}" failed. No retries permitted until 2026-09-07 10:11:38.493770155 +0000 UTC m=+5.042285123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8szd" (UniqueName: "kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd") pod "network-check-target-zzfpd" (UID: "98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:36.806014 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.805872    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 10:06:34 +0000 UTC" deadline="2028-05-18 19:40:21.943249942 +0000 UTC"
Sep 07 10:11:36.806014 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.805928    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14865h28m45.137326174s"
Sep 07 10:11:36.949510 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:36.949301    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 10:11:37.893375 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:37.892198    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:37.893375 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:37.892313    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:37.893375 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:37.892805    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:37.893375 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:37.892907    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:37.894376 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:37.892063    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:37.894376 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:37.894263    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:38.508423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:38.508726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:38.508590    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:38.508761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:38.508820    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs podName:232a763f-6ba8-4484-855b-ead84890b628 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:42.508797594 +0000 UTC m=+9.057312579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs") pod "network-metrics-daemon-dbjml" (UID: "232a763f-6ba8-4484-855b-ead84890b628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:38.508877    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:38.508927    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret podName:bfd233c5-b9bd-47b1-aee1-1013a76983b5 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:42.508911062 +0000 UTC m=+9.057426033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret") pod "global-pull-secret-syncer-jr9hg" (UID: "bfd233c5-b9bd-47b1-aee1-1013a76983b5") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:38.508924    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:38.508951    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:38.508964    2560 projected.go:196] Error preparing data for projected volume kube-api-access-s8szd for pod openshift-network-diagnostics/network-check-target-zzfpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:38.509030 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:38.509005    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd podName:98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d nodeName:}" failed. No retries permitted until 2026-09-07 10:11:42.508993778 +0000 UTC m=+9.057508746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8szd" (UniqueName: "kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd") pod "network-check-target-zzfpd" (UID: "98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:39.889469 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:39.889438    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:39.889919 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:39.889570    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:39.889919 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:39.889648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:39.889919 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:39.889762    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:39.889919 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:39.889438    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:39.889919 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:39.889853    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:41.889972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:41.889925    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:41.889972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:41.889925    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:41.889972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:41.889925    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:41.890488 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:41.890095    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:41.890488 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:41.890259    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:41.890488 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:41.890342    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:42.540454 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:42.540416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:42.540645 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:42.540474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:42.540645 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:42.540507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:42.540645 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:42.540567    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:42.540645 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:42.540604    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:42.540645 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:42.540634    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs podName:232a763f-6ba8-4484-855b-ead84890b628 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:50.540616175 +0000 UTC m=+17.089131164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs") pod "network-metrics-daemon-dbjml" (UID: "232a763f-6ba8-4484-855b-ead84890b628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:42.540645 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:42.540633    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:42.540645 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:42.540650    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret podName:bfd233c5-b9bd-47b1-aee1-1013a76983b5 nodeName:}" failed. No retries permitted until 2026-09-07 10:11:50.540643588 +0000 UTC m=+17.089158553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret") pod "global-pull-secret-syncer-jr9hg" (UID: "bfd233c5-b9bd-47b1-aee1-1013a76983b5") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:42.540975 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:42.540662    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:42.540975 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:42.540675    2560 projected.go:196] Error preparing data for projected volume kube-api-access-s8szd for pod openshift-network-diagnostics/network-check-target-zzfpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:42.540975 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:42.540718    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd podName:98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d nodeName:}" failed. No retries permitted until 2026-09-07 10:11:50.540704187 +0000 UTC m=+17.089219167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8szd" (UniqueName: "kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd") pod "network-check-target-zzfpd" (UID: "98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:43.892681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:43.892653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:43.893092 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:43.892759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:43.893092 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:43.892768    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:43.893092 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:43.892776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:43.893092 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:43.892878    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:43.893092 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:43.892942    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:45.891957 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:45.891933    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:45.892348 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:45.891931    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:45.892348 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:45.892049    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:45.892348 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:45.891931    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:45.892348 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:45.892116    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:45.892348 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:45.892156    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:47.890114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:47.890079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:47.890544 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:47.890092    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:47.890544 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:47.890191    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:47.890544 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:47.890210    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:47.890544 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:47.890326    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:47.890544 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:47.890409    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:49.890005 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:49.889976    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:49.890005 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:49.890002    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:49.890548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:49.889978    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:49.890548 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:49.890110    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:49.890548 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:49.890218    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:49.890548 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:49.890284    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:50.599040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:50.599005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:50.599063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:50.599099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:50.599144    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:50.599202    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:50.599212    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs podName:232a763f-6ba8-4484-855b-ead84890b628 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:06.599191642 +0000 UTC m=+33.147706621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs") pod "network-metrics-daemon-dbjml" (UID: "232a763f-6ba8-4484-855b-ead84890b628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:50.599218    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:50.599230    2560 projected.go:196] Error preparing data for projected volume kube-api-access-s8szd for pod openshift-network-diagnostics/network-check-target-zzfpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:50.599217    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:50.599264    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd podName:98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d nodeName:}" failed. No retries permitted until 2026-09-07 10:12:06.599253652 +0000 UTC m=+33.147768620 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8szd" (UniqueName: "kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd") pod "network-check-target-zzfpd" (UID: "98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:11:50.599274 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:50.599281    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret podName:bfd233c5-b9bd-47b1-aee1-1013a76983b5 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:06.599272064 +0000 UTC m=+33.147787028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret") pod "global-pull-secret-syncer-jr9hg" (UID: "bfd233c5-b9bd-47b1-aee1-1013a76983b5") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:11:51.889699 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:51.889664    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:51.890243 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:51.889673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:51.890243 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:51.889804    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:51.890243 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:51.889851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:51.890243 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:51.889937    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:51.890243 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:51.889986    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:53.890012 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:53.889982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:53.890332 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:53.890102    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:53.890332 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:53.890185    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:53.890332 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:53.890097    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:53.890332 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:53.890239    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:53.890332 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:53.890315    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:55.132569 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.132197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cjhxw" event={"ID":"679b04d9-df87-45a9-8bc1-8cb7d5c88758","Type":"ContainerStarted","Data":"0d83313854e7f3120ed71bf0c6de392a6712b10e2fb51e7a74e75e90ca8c73af"}
Sep 07 10:11:55.134161 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.134131    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ce3557a-cc31-4782-9785-ed93485f273a" containerID="3574cf14165fd82118bf56ddae8e85de0a1a28f5c50577498a157ac796f1fc41" exitCode=0
Sep 07 10:11:55.134265 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.134205    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shht6" event={"ID":"1ce3557a-cc31-4782-9785-ed93485f273a","Type":"ContainerDied","Data":"3574cf14165fd82118bf56ddae8e85de0a1a28f5c50577498a157ac796f1fc41"}
Sep 07 10:11:55.135839 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.135810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxqmk" event={"ID":"969e1cf8-0598-41cd-a6d5-626d5fc54ea7","Type":"ContainerStarted","Data":"ecdce9d6f5eb1b59423dfd133bce06d1d5335317822dc33714078ef6d986590b"}
Sep 07 10:11:55.139222 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.139197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"44c62649f2ca677310e40c06ba1c416b5306cca316e4164607b8befae83a18d0"}
Sep 07 10:11:55.139320 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.139272    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"b6101221bc9bc337dbfb473ebd7882847aa5afeffc33662d459ad99bc97d495a"}
Sep 07 10:11:55.139384 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.139325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"2c4dadef2f600afd6cac9eaff51e10e9afd035f29e7ef1e9ca70872896d50e20"}
Sep 07 10:11:55.139384 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.139374    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"ca3e76cb6b02e7bd6c614524a7911250f6582fa26297db0ebb694b0c5349bde2"}
Sep 07 10:11:55.139484 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.139424    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"b94c0a64f8873afe1689172ee5c1a196242d6e90e2884864c1a9f9de7220a05b"}
Sep 07 10:11:55.139484 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.139471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"415786b7af58fccacfd18dafb33df48da529b46d14eafb76fdb4473ef84a9cfb"}
Sep 07 10:11:55.140907 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.140885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx" event={"ID":"0e67e468-8cf2-4095-8878-e5698eaa805a","Type":"ContainerStarted","Data":"9fe160d5871002b89370745551d639e2eddad879a55b40d9e21f5551f8475e9c"}
Sep 07 10:11:55.142824 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.142796    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9f25" event={"ID":"5d2505c0-2b3f-41c8-a8e8-ebf813ba8d46","Type":"ContainerStarted","Data":"3d23d3d3c052b862025d93bec59f5fab21d8e2100608f655cccd5558cb6724c5"}
Sep 07 10:11:55.144312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.144285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" event={"ID":"c349833f5e025366a333ee7959fbf2b3","Type":"ContainerStarted","Data":"ea25f5acd40f09c79489561edbc2ff7580c7ffac3f70d1229485d73f0ddc10ef"}
Sep 07 10:11:55.145899 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.145875    2560 generic.go:356] "Generic (PLEG): container finished" podID="c6acb16e1cdb3c44a9ba2958443fb258" containerID="3f19ca0fb3e8bdeee39d75a39905d80a7359ddbf28f53e98a02d848a3d984fe6" exitCode=0
Sep 07 10:11:55.145990 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.145944    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" event={"ID":"c6acb16e1cdb3c44a9ba2958443fb258","Type":"ContainerDied","Data":"3f19ca0fb3e8bdeee39d75a39905d80a7359ddbf28f53e98a02d848a3d984fe6"}
Sep 07 10:11:55.146536 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.146498    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cjhxw" podStartSLOduration=3.560286112 podStartE2EDuration="22.146486962s" totalImagesPullingTime="18.58620085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:55.146394791 +0000 UTC m=+21.694909779" watchObservedRunningTime="2026-09-07 10:11:55.146486962 +0000 UTC m=+21.695001946"
Sep 07 10:11:55.152702 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.151404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-56glg" event={"ID":"eb2119b6-8f00-4d3d-a2f8-3689940c55fd","Type":"ContainerStarted","Data":"e65c05eecb73824057d48263532a720111c4e2341936d3f0c6503f716f65b39b"}
Sep 07 10:11:55.153861 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.153833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ngwlm" event={"ID":"c50d79e9-af4e-4927-b4f4-2cfd83f3290e","Type":"ContainerStarted","Data":"50445da90d47ee3e1c6898a08c6feb024824630bd73ec136c5c25a6cf960d04f"}
Sep 07 10:11:55.176943 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.176900    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-191.ec2.internal" podStartSLOduration=21.176885287 podStartE2EDuration="21.176885287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:55.158907991 +0000 UTC m=+21.707422977" watchObservedRunningTime="2026-09-07 10:11:55.176885287 +0000 UTC m=+21.725400301"
Sep 07 10:11:55.190721 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.190686    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w9f25" podStartSLOduration=3.189888534 podStartE2EDuration="22.190672939s" totalImagesPullingTime="19.000784405s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:55.190570285 +0000 UTC m=+21.739085273" watchObservedRunningTime="2026-09-07 10:11:55.190672939 +0000 UTC m=+21.739188005"
Sep 07 10:11:55.205937 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.205891    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sxqmk" podStartSLOduration=2.3136498850000002 podStartE2EDuration="21.205871269s" totalImagesPullingTime="18.892221384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:55.205517853 +0000 UTC m=+21.754032840" watchObservedRunningTime="2026-09-07 10:11:55.205871269 +0000 UTC m=+21.754386257"
Sep 07 10:11:55.225374 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.225337    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-56glg" podStartSLOduration=2.315809961 podStartE2EDuration="21.22532369s" totalImagesPullingTime="18.909513729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:55.225121523 +0000 UTC m=+21.773636510" watchObservedRunningTime="2026-09-07 10:11:55.22532369 +0000 UTC m=+21.773838768"
Sep 07 10:11:55.240756 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.240725    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ngwlm" podStartSLOduration=3.222722282 podStartE2EDuration="22.240713694s" totalImagesPullingTime="19.017991412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:55.240686969 +0000 UTC m=+21.789201955" watchObservedRunningTime="2026-09-07 10:11:55.240713694 +0000 UTC m=+21.789228680"
Sep 07 10:11:55.261705 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.261676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:55.345330 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.345292    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:55.445017 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.444960    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:55.545454 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.545426    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:55.890198 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.890172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:55.890316 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.890188    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:55.890374 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:55.890306    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:55.890434 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:55.890377    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:55.890490 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.890436    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:55.890539 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:55.890515    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:55.939706 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:55.939609    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 10:11:56.158315 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.158224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hjfhg" event={"ID":"ffd270e7-a91f-41cd-918e-c2650aa8f280","Type":"ContainerStarted","Data":"4dbbfdad8bde557685a9edb6dd87e524353662e2db79591f2a6020ccf28807bc"}
Sep 07 10:11:56.160434 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.160400    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx" event={"ID":"0e67e468-8cf2-4095-8878-e5698eaa805a","Type":"ContainerStarted","Data":"356ae520fd705ba4da0271873f06581ff548cc69c698092cb4ca0478e29d1ac9"}
Sep 07 10:11:56.174515 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.174375    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hjfhg" podStartSLOduration=4.143502152 podStartE2EDuration="23.174358684s" totalImagesPullingTime="19.030856532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:56.173575537 +0000 UTC m=+22.722090552" watchObservedRunningTime="2026-09-07 10:11:56.174358684 +0000 UTC m=+22.722873672"
Sep 07 10:11:56.263871 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.263841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:56.360034 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.359988    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:56.417196 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.417120    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:56.817559 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.817449    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T10:11:55.939625673Z","UUID":"64e5a935-b456-4106-90c7-a271eed87eec","Handler":null,"Name":"","Endpoint":""}
Sep 07 10:11:56.819554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.819533    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 10:11:56.819674 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.819565    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 10:11:56.989245 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:56.989206    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:57.092737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.092648    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:57.164959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.164932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx" event={"ID":"0e67e468-8cf2-4095-8878-e5698eaa805a","Type":"ContainerStarted","Data":"2def01a93908b2557070c89fbf261356ecb2e6d5f6a2795f7056d24bb65687fb"}
Sep 07 10:11:57.167195 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.167118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" event={"ID":"c6acb16e1cdb3c44a9ba2958443fb258","Type":"ContainerStarted","Data":"ba16dd6dec1b440f13326f965226b255b5860e7688a1baa5888e0d3151a1cb58"}
Sep 07 10:11:57.181352 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.181305    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmqfx" podStartSLOduration=2.515592259 podStartE2EDuration="23.181289492s" totalImagesPullingTime="20.665697233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:57.181233545 +0000 UTC m=+23.729748532" watchObservedRunningTime="2026-09-07 10:11:57.181289492 +0000 UTC m=+23.729804480"
Sep 07 10:11:57.704530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.704495    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:57.762752 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.762719    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:57.890104 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.889942    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:57.890253 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.889942    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:57.890253 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:57.890199    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:11:57.890363 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:57.890290    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:57.890363 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:57.889980    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:57.890458 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:57.890398    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:58.173110 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:58.172982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"e1e8dc47768bcbd998bbf0dc81f0da44b56e86e6dbc7d72d842c83395302408f"}
Sep 07 10:11:59.273947 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:59.273905    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:59.374219 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:59.374185    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:11:59.889340 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:59.889141    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:11:59.889441 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:59.889141    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:11:59.889441 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:59.889392    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:11:59.889509 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:11:59.889195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:11:59.889542 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:59.889450    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:11:59.889542 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:11:59.889530    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:12:00.086992 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.086969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:12:00.087623 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.087609    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:12:00.101319 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.101283    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-191.ec2.internal" podStartSLOduration=26.101272053 podStartE2EDuration="26.101272053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:11:57.195729239 +0000 UTC m=+23.744244226" watchObservedRunningTime="2026-09-07 10:12:00.101272053 +0000 UTC m=+26.649787040"
Sep 07 10:12:00.178708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.178656    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ce3557a-cc31-4782-9785-ed93485f273a" containerID="fd93f1973ed76cddbc962415fff723de7d7f57724a648c25169bc05e5644b802" exitCode=0
Sep 07 10:12:00.178805 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.178727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shht6" event={"ID":"1ce3557a-cc31-4782-9785-ed93485f273a","Type":"ContainerDied","Data":"fd93f1973ed76cddbc962415fff723de7d7f57724a648c25169bc05e5644b802"}
Sep 07 10:12:00.182613 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.182594    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" event={"ID":"b57956ef-2740-4da5-b4ce-78e476de56e9","Type":"ContainerStarted","Data":"758e5e349e68ab5cdbec5a27cdd70b1c1b1fb36dd2d4c1c4e95ad829b68ae7bf"}
Sep 07 10:12:00.182829 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.182808    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:12:00.182924 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.182857    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:12:00.200415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.200379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:12:00.235468 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.235430    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66" podStartSLOduration=6.944297155 podStartE2EDuration="26.235417776s" totalImagesPullingTime="19.291120621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:00.233970204 +0000 UTC m=+26.782485187" watchObservedRunningTime="2026-09-07 10:12:00.235417776 +0000 UTC m=+26.783932762"
Sep 07 10:12:00.250186 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.250166    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:00.310824 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:00.310804    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:01.187100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.187059    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ce3557a-cc31-4782-9785-ed93485f273a" containerID="4071093b86af3a43cb1d8246d00b86729f35ef7523d8526220a26e24c300c286" exitCode=0
Sep 07 10:12:01.187360 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.187113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shht6" event={"ID":"1ce3557a-cc31-4782-9785-ed93485f273a","Type":"ContainerDied","Data":"4071093b86af3a43cb1d8246d00b86729f35ef7523d8526220a26e24c300c286"}
Sep 07 10:12:01.187633 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.187614    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:12:01.201320 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.201298    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:12:01.379284 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.379222    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zzfpd"]
Sep 07 10:12:01.379922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.379343    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:01.379922 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:01.379436    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:12:01.382769 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.382744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dbjml"]
Sep 07 10:12:01.382875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.382846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:01.382944 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:01.382924    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:12:01.383428 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.383404    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jr9hg"]
Sep 07 10:12:01.383508 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:01.383499    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:01.383607 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:01.383587    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:12:02.187521 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:02.187442    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:02.191153 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:02.191129    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ce3557a-cc31-4782-9785-ed93485f273a" containerID="0ea957760ff4a0843200a3389d4e8aa82848bf269126968febc66583222ab02d" exitCode=0
Sep 07 10:12:02.191262 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:02.191182    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shht6" event={"ID":"1ce3557a-cc31-4782-9785-ed93485f273a","Type":"ContainerDied","Data":"0ea957760ff4a0843200a3389d4e8aa82848bf269126968febc66583222ab02d"}
Sep 07 10:12:02.889178 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:02.889112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:02.889513 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:02.889114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:02.889513 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:02.889114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:02.889513 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:02.889278    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:12:02.889513 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:02.889347    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:12:02.889513 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:02.889193    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:12:03.001446 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:03.001425    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:12:03.001560 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:03.001522    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 07 10:12:03.002116 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:03.002095    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sxqmk"
Sep 07 10:12:03.061712 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:03.061626    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:04.890130 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:04.889921    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:04.890517 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:04.889975    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:04.890517 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:04.890199    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:12:04.890517 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:04.889975    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:04.890517 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:04.890283    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:12:04.890517 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:04.890364    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:12:05.045742 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:05.045714    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:05.146918 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:05.146848    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:05.438273 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:05.438184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:05.539252 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:05.539217    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:05.591304 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:05.591275    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:06.626040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:06.626009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:06.626051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.626181    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.626195    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.626218    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.626227    2560 projected.go:196] Error preparing data for projected volume kube-api-access-s8szd for pod openshift-network-diagnostics/network-check-target-zzfpd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:06.626255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.626277    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd podName:98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d nodeName:}" failed. No retries permitted until 2026-09-07 10:12:38.6262591 +0000 UTC m=+65.174774080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s8szd" (UniqueName: "kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd") pod "network-check-target-zzfpd" (UID: "98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.626302    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs podName:232a763f-6ba8-4484-855b-ead84890b628 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:38.62629223 +0000 UTC m=+65.174807195 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs") pod "network-metrics-daemon-dbjml" (UID: "232a763f-6ba8-4484-855b-ead84890b628") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.626341    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 10:12:06.626528 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.626393    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret podName:bfd233c5-b9bd-47b1-aee1-1013a76983b5 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:38.626377361 +0000 UTC m=+65.174892352 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret") pod "global-pull-secret-syncer-jr9hg" (UID: "bfd233c5-b9bd-47b1-aee1-1013a76983b5") : object "kube-system"/"original-pull-secret" not registered
Sep 07 10:12:06.889184 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:06.889108    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:06.889184 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:06.889128    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:06.889467 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:06.889120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:06.889467 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.889235    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9hg" podUID="bfd233c5-b9bd-47b1-aee1-1013a76983b5"
Sep 07 10:12:06.889467 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.889384    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zzfpd" podUID="98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d"
Sep 07 10:12:06.889615 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:06.889492    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dbjml" podUID="232a763f-6ba8-4484-855b-ead84890b628"
Sep 07 10:12:07.238516 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.238450    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-191.ec2.internal" event="NodeReady"
Sep 07 10:12:07.238758 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.238562    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 10:12:07.273537 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.273509    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58c774d8d7-m4f7f"]
Sep 07 10:12:07.282397 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.282377    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.283461 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.283433    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jq7l9"]
Sep 07 10:12:07.284703 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.284682    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 07 10:12:07.284817 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.284714    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 07 10:12:07.284990 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.284974    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hpk4\""
Sep 07 10:12:07.285109 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.285093    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 07 10:12:07.290163 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.289922    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 07 10:12:07.294531 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.294512    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nj8x8"]
Sep 07 10:12:07.294638 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.294628    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jq7l9"
Sep 07 10:12:07.297210 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.297189    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 10:12:07.297310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.297219    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 10:12:07.297310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.297219    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gq9jw\""
Sep 07 10:12:07.297725 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.297702    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 10:12:07.318812 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.318791    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58c774d8d7-m4f7f"]
Sep 07 10:12:07.318900 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.318818    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jq7l9"]
Sep 07 10:12:07.318900 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.318832    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nj8x8"]
Sep 07 10:12:07.318900 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.318898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.320992 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.320976    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 10:12:07.321209 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.321186    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 10:12:07.321209 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.321202    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8d2\""
Sep 07 10:12:07.420509 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.420482    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:07.432370 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-bound-sa-token\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.432469 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgn84\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-kube-api-access-sgn84\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.432469 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjhw\" (UniqueName: \"kubernetes.io/projected/4ab0827d-36bd-4e44-b152-b72cf5cf8861-kube-api-access-4hjhw\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.432469 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.432571 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.432571 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-registry-certificates\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.432628 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d685669b-5eb2-4358-a543-f59a8df88f4e-cert\") pod \"ingress-canary-jq7l9\" (UID: \"d685669b-5eb2-4358-a543-f59a8df88f4e\") " pod="openshift-ingress-canary/ingress-canary-jq7l9"
Sep 07 10:12:07.432628 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/889199f9-100b-4bb5-86a1-66584c032c22-ca-trust-extracted\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.432628 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432606    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-installation-pull-secrets\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.432717 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-image-registry-private-configuration\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.432717 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-trusted-ca\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.432717 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab0827d-36bd-4e44-b152-b72cf5cf8861-config-volume\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.432717 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab0827d-36bd-4e44-b152-b72cf5cf8861-tmp-dir\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.432833 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.432720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwwjr\" (UniqueName: \"kubernetes.io/projected/d685669b-5eb2-4358-a543-f59a8df88f4e-kube-api-access-zwwjr\") pod \"ingress-canary-jq7l9\" (UID: \"d685669b-5eb2-4358-a543-f59a8df88f4e\") " pod="openshift-ingress-canary/ingress-canary-jq7l9"
Sep 07 10:12:07.533074 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwwjr\" (UniqueName: \"kubernetes.io/projected/d685669b-5eb2-4358-a543-f59a8df88f4e-kube-api-access-zwwjr\") pod \"ingress-canary-jq7l9\" (UID: \"d685669b-5eb2-4358-a543-f59a8df88f4e\") " pod="openshift-ingress-canary/ingress-canary-jq7l9"
Sep 07 10:12:07.533175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-bound-sa-token\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.533175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgn84\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-kube-api-access-sgn84\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.533175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjhw\" (UniqueName: \"kubernetes.io/projected/4ab0827d-36bd-4e44-b152-b72cf5cf8861-kube-api-access-4hjhw\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.533175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533169    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.533374 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.533374 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-registry-certificates\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.533374 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d685669b-5eb2-4358-a543-f59a8df88f4e-cert\") pod \"ingress-canary-jq7l9\" (UID: \"d685669b-5eb2-4358-a543-f59a8df88f4e\") " pod="openshift-ingress-canary/ingress-canary-jq7l9"
Sep 07 10:12:07.533374 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/889199f9-100b-4bb5-86a1-66584c032c22-ca-trust-extracted\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.533374 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:07.533327    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-installation-pull-secrets\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:07.533332    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:07.533416    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c774d8d7-m4f7f: secret "image-registry-tls" not found
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:07.533419    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls podName:4ab0827d-36bd-4e44-b152-b72cf5cf8861 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:08.033388226 +0000 UTC m=+34.581903205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls") pod "dns-default-nj8x8" (UID: "4ab0827d-36bd-4e44-b152-b72cf5cf8861") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:07.533465    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls podName:889199f9-100b-4bb5-86a1-66584c032c22 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:08.033453082 +0000 UTC m=+34.581968067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls") pod "image-registry-58c774d8d7-m4f7f" (UID: "889199f9-100b-4bb5-86a1-66584c032c22") : secret "image-registry-tls" not found
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-image-registry-private-configuration\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-trusted-ca\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab0827d-36bd-4e44-b152-b72cf5cf8861-config-volume\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.533609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab0827d-36bd-4e44-b152-b72cf5cf8861-tmp-dir\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.534029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/889199f9-100b-4bb5-86a1-66584c032c22-ca-trust-extracted\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.534029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533874    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ab0827d-36bd-4e44-b152-b72cf5cf8861-tmp-dir\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.534029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.533881    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-registry-certificates\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.534297 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.534277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-trusted-ca\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.534347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.534302    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab0827d-36bd-4e44-b152-b72cf5cf8861-config-volume\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.537435 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.537409    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-installation-pull-secrets\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.537537 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.537493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d685669b-5eb2-4358-a543-f59a8df88f4e-cert\") pod \"ingress-canary-jq7l9\" (UID: \"d685669b-5eb2-4358-a543-f59a8df88f4e\") " pod="openshift-ingress-canary/ingress-canary-jq7l9"
Sep 07 10:12:07.537537 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.537505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-image-registry-private-configuration\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.544426 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.544399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgn84\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-kube-api-access-sgn84\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.544426 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.544411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-bound-sa-token\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:07.545000 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.544982    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjhw\" (UniqueName: \"kubernetes.io/projected/4ab0827d-36bd-4e44-b152-b72cf5cf8861-kube-api-access-4hjhw\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:07.552570 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.552551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwwjr\" (UniqueName: \"kubernetes.io/projected/d685669b-5eb2-4358-a543-f59a8df88f4e-kube-api-access-zwwjr\") pod \"ingress-canary-jq7l9\" (UID: \"d685669b-5eb2-4358-a543-f59a8df88f4e\") " pod="openshift-ingress-canary/ingress-canary-jq7l9"
Sep 07 10:12:07.605162 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.605145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jq7l9"
Sep 07 10:12:07.917523 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.917356    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jq7l9"]
Sep 07 10:12:07.922794 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:07.922761    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd685669b_5eb2_4358_a543_f59a8df88f4e.slice/crio-39c1f877a7ce5c383f7a41bbf1836027d9b2eb951062e0203dd2002342046d9c WatchSource:0}: Error finding container 39c1f877a7ce5c383f7a41bbf1836027d9b2eb951062e0203dd2002342046d9c: Status 404 returned error can't find the container with id 39c1f877a7ce5c383f7a41bbf1836027d9b2eb951062e0203dd2002342046d9c
Sep 07 10:12:07.973631 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:07.973602    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:08.036609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.036583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:08.036694 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.036617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:08.036732 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:08.036723    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:08.036777 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:08.036766    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:12:08.036810 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:08.036776    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls podName:4ab0827d-36bd-4e44-b152-b72cf5cf8861 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:09.036762528 +0000 UTC m=+35.585277493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls") pod "dns-default-nj8x8" (UID: "4ab0827d-36bd-4e44-b152-b72cf5cf8861") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:08.036810 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:08.036778    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c774d8d7-m4f7f: secret "image-registry-tls" not found
Sep 07 10:12:08.036876 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:08.036816    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls podName:889199f9-100b-4bb5-86a1-66584c032c22 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:09.036805414 +0000 UTC m=+35.585320380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls") pod "image-registry-58c774d8d7-m4f7f" (UID: "889199f9-100b-4bb5-86a1-66584c032c22") : secret "image-registry-tls" not found
Sep 07 10:12:08.074057 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.074024    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:08.207348 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.207294    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ce3557a-cc31-4782-9785-ed93485f273a" containerID="adcdab750374cacb6bd47f7d76debf16ae84fc6e37b2fd0fc0189f340a9cc52e" exitCode=0
Sep 07 10:12:08.207432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.207357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shht6" event={"ID":"1ce3557a-cc31-4782-9785-ed93485f273a","Type":"ContainerDied","Data":"adcdab750374cacb6bd47f7d76debf16ae84fc6e37b2fd0fc0189f340a9cc52e"}
Sep 07 10:12:08.208596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.208574    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jq7l9" event={"ID":"d685669b-5eb2-4358-a543-f59a8df88f4e","Type":"ContainerStarted","Data":"39c1f877a7ce5c383f7a41bbf1836027d9b2eb951062e0203dd2002342046d9c"}
Sep 07 10:12:08.889127 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.889095    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:08.889334 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.889100    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:08.889334 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.889100    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:08.891883 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.891858    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 10:12:08.892318 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.892259    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9xn8\""
Sep 07 10:12:08.892514 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.892501    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 10:12:08.892865 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.892849    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 10:12:08.892955 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.892936    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cp5xp\""
Sep 07 10:12:08.915363 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:08.915293    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 10:12:09.044020 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:09.043878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:09.044020 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:09.043929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:09.044522 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:09.044049    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:09.044522 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:09.044131    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls podName:4ab0827d-36bd-4e44-b152-b72cf5cf8861 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:11.044112513 +0000 UTC m=+37.592627493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls") pod "dns-default-nj8x8" (UID: "4ab0827d-36bd-4e44-b152-b72cf5cf8861") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:09.044522 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:09.044136    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:12:09.044522 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:09.044151    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c774d8d7-m4f7f: secret "image-registry-tls" not found
Sep 07 10:12:09.044522 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:09.044225    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls podName:889199f9-100b-4bb5-86a1-66584c032c22 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:11.044208228 +0000 UTC m=+37.592723208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls") pod "image-registry-58c774d8d7-m4f7f" (UID: "889199f9-100b-4bb5-86a1-66584c032c22") : secret "image-registry-tls" not found
Sep 07 10:12:09.214910 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:09.214823    2560 generic.go:356] "Generic (PLEG): container finished" podID="1ce3557a-cc31-4782-9785-ed93485f273a" containerID="4d92ea784751037cf3c70407ad4e7e58016465735a1d7081ac4b3805aaa4d228" exitCode=0
Sep 07 10:12:09.214910 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:09.214870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shht6" event={"ID":"1ce3557a-cc31-4782-9785-ed93485f273a","Type":"ContainerDied","Data":"4d92ea784751037cf3c70407ad4e7e58016465735a1d7081ac4b3805aaa4d228"}
Sep 07 10:12:10.219895 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:10.219863    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-shht6" event={"ID":"1ce3557a-cc31-4782-9785-ed93485f273a","Type":"ContainerStarted","Data":"44ff8a9dfa3b9af071645e88636846073b05cfe3a47c9278451e1828e7fc9c55"}
Sep 07 10:12:10.241949 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:10.241899    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-shht6" podStartSLOduration=7.302220437 podStartE2EDuration="37.241883685s" totalImagesPullingTime="29.939663248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:10.24047876 +0000 UTC m=+36.788993747" watchObservedRunningTime="2026-09-07 10:12:10.241883685 +0000 UTC m=+36.790398671"
Sep 07 10:12:10.361023 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:10.360996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:10.461935 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:10.461885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:11.059312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:11.059280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:11.059455 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:11.059325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:11.059455 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:11.059436    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:11.059530 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:11.059495    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls podName:4ab0827d-36bd-4e44-b152-b72cf5cf8861 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.059481372 +0000 UTC m=+41.607996350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls") pod "dns-default-nj8x8" (UID: "4ab0827d-36bd-4e44-b152-b72cf5cf8861") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:11.059530 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:11.059440    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:12:11.059616 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:11.059536    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c774d8d7-m4f7f: secret "image-registry-tls" not found
Sep 07 10:12:11.059616 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:11.059573    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls podName:889199f9-100b-4bb5-86a1-66584c032c22 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.059562733 +0000 UTC m=+41.608077698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls") pod "image-registry-58c774d8d7-m4f7f" (UID: "889199f9-100b-4bb5-86a1-66584c032c22") : secret "image-registry-tls" not found
Sep 07 10:12:11.223312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:11.223223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jq7l9" event={"ID":"d685669b-5eb2-4358-a543-f59a8df88f4e","Type":"ContainerStarted","Data":"543cc075aa7e771cd3069aca95e6a415c75c26dd7d281990f9be434a21e15987"}
Sep 07 10:12:11.239546 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:11.239504    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jq7l9" podStartSLOduration=1.862912428 podStartE2EDuration="4.239490547s" totalImagesPullingTime="2.376578119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:11.238026197 +0000 UTC m=+37.786541185" watchObservedRunningTime="2026-09-07 10:12:11.239490547 +0000 UTC m=+37.788005527"
Sep 07 10:12:11.495465 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:11.495391    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:11.595866 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:11.595838    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:12.368401 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:12.368363    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:12.603760 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:12.603730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:12.694192 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:12.694103    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:13.090663 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.090625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:13.192396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.192350    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:13.473185 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.473115    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6ft8s"]
Sep 07 10:12:13.485764 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.485731    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:13.490007 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.489984    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6ft8s"]
Sep 07 10:12:13.490121 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.490108    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.492714 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.492694    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gp5c5\""
Sep 07 10:12:13.492991 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.492965    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 10:12:13.493196 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.493182    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 10:12:13.493246 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.493219    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 10:12:13.493278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.493251    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 10:12:13.543700 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.543678    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:13.648611 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.648584    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:13.676577 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.676555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgjw\" (UniqueName: \"kubernetes.io/projected/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-api-access-bfgjw\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.676676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.676582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89b831f6-f4b9-4ddf-baed-806b217f0c8e-crio-socket\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.676676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.676603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89b831f6-f4b9-4ddf-baed-806b217f0c8e-data-volume\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.676676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.676619    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.676676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.676649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.744664 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.744595    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:13.777583 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.777555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgjw\" (UniqueName: \"kubernetes.io/projected/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-api-access-bfgjw\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.777702 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.777585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89b831f6-f4b9-4ddf-baed-806b217f0c8e-crio-socket\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.777702 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.777602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89b831f6-f4b9-4ddf-baed-806b217f0c8e-data-volume\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.777702 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.777617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.777702 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.777656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.777893 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:13.777738    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:13.777893 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:13.777769    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:13.777893 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.777785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/89b831f6-f4b9-4ddf-baed-806b217f0c8e-crio-socket\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.777893 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:13.777800    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:14.277783164 +0000 UTC m=+40.826298145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:13.777893 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:13.777825    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:14.277808355 +0000 UTC m=+40.826323321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:13.778120 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.777929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/89b831f6-f4b9-4ddf-baed-806b217f0c8e-data-volume\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:13.788925 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:13.788899    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgjw\" (UniqueName: \"kubernetes.io/projected/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-api-access-bfgjw\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:14.281348 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:14.281315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:14.281588 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:14.281377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:14.281588 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:14.281453    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:14.281588 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:14.281506    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:14.281588 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:14.281537    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.281521611 +0000 UTC m=+41.830036576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:14.281588 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:14.281566    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:15.281553075 +0000 UTC m=+41.830068066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:15.087444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:15.087408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:15.087845 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:15.087450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:15.087845 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.087549    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:15.087845 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.087560    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:12:15.087845 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.087576    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c774d8d7-m4f7f: secret "image-registry-tls" not found
Sep 07 10:12:15.087845 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.087602    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls podName:4ab0827d-36bd-4e44-b152-b72cf5cf8861 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:23.087586741 +0000 UTC m=+49.636101706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls") pod "dns-default-nj8x8" (UID: "4ab0827d-36bd-4e44-b152-b72cf5cf8861") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:15.087845 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.087624    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls podName:889199f9-100b-4bb5-86a1-66584c032c22 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:23.087609859 +0000 UTC m=+49.636124841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls") pod "image-registry-58c774d8d7-m4f7f" (UID: "889199f9-100b-4bb5-86a1-66584c032c22") : secret "image-registry-tls" not found
Sep 07 10:12:15.289354 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:15.289323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:15.289489 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.289440    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:15.289489 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:15.289472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:15.289580 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.289494    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:17.28947844 +0000 UTC m=+43.837993405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:15.289580 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.289538    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:15.289692 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:15.289582    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:17.289570097 +0000 UTC m=+43.838085062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:15.320505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:15.320474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:15.897168 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:15.897136    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:16.012716 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:16.012683    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:16.035542 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:16.035517    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjhxw_679b04d9-df87-45a9-8bc1-8cb7d5c88758/dns-node-resolver/0.log"
Sep 07 10:12:16.357419 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:16.357388    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:17.033642 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:17.033616    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-56glg_eb2119b6-8f00-4d3d-a2f8-3689940c55fd/node-ca/0.log"
Sep 07 10:12:17.307229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:17.307159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:17.307229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:17.307203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:17.307409 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:17.307270    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:17.307409 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:17.307291    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:17.307409 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:17.307320    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.30730686 +0000 UTC m=+47.855821825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:17.307409 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:17.307337    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:21.307324915 +0000 UTC m=+47.855839898 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:18.027673 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:18.027644    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jq7l9_d685669b-5eb2-4358-a543-f59a8df88f4e/serve-healthcheck-canary/0.log"
Sep 07 10:12:19.317805 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:19.317760    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:20.231692 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:20.231665    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:20.426172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:20.426137    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:20.527246 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:20.527153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:20.630241 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:20.629862    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:20.728028 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:20.727994    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:20.826881 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:20.826841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:20.927030 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:20.927001    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:21.334800 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:21.334775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:21.334952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:21.334822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:21.334952 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:21.334929    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:21.335099 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:21.334954    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 10:12:21.335099 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:21.334996    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:29.334981291 +0000 UTC m=+55.883496260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:21.335099 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:21.335015    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:29.335001707 +0000 UTC m=+55.883516671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : configmap "kube-rbac-proxy" not found
Sep 07 10:12:21.947939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:21.947909    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:22.320371 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:22.320331    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:22.521453 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:22.521418    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:22.638290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:22.638199    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:22.739318 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:22.739275    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:23.124383 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:23.124342    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:23.150195 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:23.150168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:23.150315 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:23.150202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:23.150315 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:23.150299    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 10:12:23.150400 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:23.150362    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls podName:4ab0827d-36bd-4e44-b152-b72cf5cf8861 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:39.150345945 +0000 UTC m=+65.698860910 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls") pod "dns-default-nj8x8" (UID: "4ab0827d-36bd-4e44-b152-b72cf5cf8861") : secret "dns-default-metrics-tls" not found
Sep 07 10:12:23.150400 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:23.150300    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 10:12:23.150400 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:23.150393    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58c774d8d7-m4f7f: secret "image-registry-tls" not found
Sep 07 10:12:23.150515 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:23.150439    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls podName:889199f9-100b-4bb5-86a1-66584c032c22 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:39.150428247 +0000 UTC m=+65.698943212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls") pod "image-registry-58c774d8d7-m4f7f" (UID: "889199f9-100b-4bb5-86a1-66584c032c22") : secret "image-registry-tls" not found
Sep 07 10:12:23.717248 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:23.717214    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:24.023736 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:24.023654    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:24.162286 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:24.162251    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:24.322530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:24.322502    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:24.523928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:24.523895    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:24.728815 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:24.728735    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:24.797677 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:24.797641    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:25.120542 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:25.120508    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:25.719135 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:25.719100    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:25.918996 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:25.918954    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:26.322280 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:26.322248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:26.505329 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:26.505291    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:26.558226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:26.558186    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:26.724205 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:26.724140    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:26.926420 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:26.926387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:27.921466 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:27.921433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:28.125377 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:28.125339    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:28.522312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:28.522270    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:28.722151 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:28.722116    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:28.923970 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:28.923940    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:29.392036 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:29.392001    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:29.392245 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:29.392152    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:29.392316 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:29.392283    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:29.392386 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:29.392369    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls podName:89b831f6-f4b9-4ddf-baed-806b217f0c8e nodeName:}" failed. No retries permitted until 2026-09-07 10:12:45.392342244 +0000 UTC m=+71.940857212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6ft8s" (UID: "89b831f6-f4b9-4ddf-baed-806b217f0c8e") : secret "insights-runtime-extractor-tls" not found
Sep 07 10:12:29.392575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:29.392552    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/89b831f6-f4b9-4ddf-baed-806b217f0c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:29.725302 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:29.725223    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:30.131411 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:30.131377    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:30.522810 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:30.522726    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:30.732410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:30.732374    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:30.832157 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:30.832134    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:30.933399 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:30.933367    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:30.989423 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:30.989386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:31.128243 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:31.128157    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:31.306286 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:31.306254    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:31.508466 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:31.506577    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:31.607005 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:31.606965    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:31.706381 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:31.706355    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:31.958982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:31.958951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:32.059207 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:32.059177    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:32.709501 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:32.709469    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:32.905527 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:32.905500    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:33.154766 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:33.154732    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:33.203030 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:33.203010    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k4r66"
Sep 07 10:12:33.722730 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:33.722701    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:33.775797 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:33.775767    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:35.044403 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:35.044357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:35.099341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:35.099316    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:38.078825 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.078795    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"]
Sep 07 10:12:38.081081 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.081050    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.084583 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.084563    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 07 10:12:38.084681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.084606    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 10:12:38.084681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.084615    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 10:12:38.084681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.084565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 10:12:38.089279 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.089256    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"]
Sep 07 10:12:38.090810 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.090795    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"
Sep 07 10:12:38.092694 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.092670    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"]
Sep 07 10:12:38.093809 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.093794    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-75khn\""
Sep 07 10:12:38.098372 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.094655    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 07 10:12:38.106084 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.106053    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"]
Sep 07 10:12:38.111229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.111210    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58c774d8d7-m4f7f"]
Sep 07 10:12:38.111410 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:38.111391    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f" podUID="889199f9-100b-4bb5-86a1-66584c032c22"
Sep 07 10:12:38.152973 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.152947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqc87\" (UniqueName: \"kubernetes.io/projected/a81c91f6-0aed-48a4-b4c7-15acb3869234-kube-api-access-qqc87\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.153053 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.152979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxc5\" (UniqueName: \"kubernetes.io/projected/8e7cc21a-e5f9-4256-8de4-3d9875a52945-kube-api-access-9qxc5\") pod \"managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5\" (UID: \"8e7cc21a-e5f9-4256-8de4-3d9875a52945\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"
Sep 07 10:12:38.153121 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.153102    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a81c91f6-0aed-48a4-b4c7-15acb3869234-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.153159 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.153135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e7cc21a-e5f9-4256-8de4-3d9875a52945-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5\" (UID: \"8e7cc21a-e5f9-4256-8de4-3d9875a52945\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"
Sep 07 10:12:38.153195 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.153172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a81c91f6-0aed-48a4-b4c7-15acb3869234-tmp\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.186840 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.186815    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6587d68c88-wz6c8"]
Sep 07 10:12:38.188558 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.188544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.205982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.205963    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6587d68c88-wz6c8"]
Sep 07 10:12:38.218775 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.218740    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:38.253557 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253537    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxc5\" (UniqueName: \"kubernetes.io/projected/8e7cc21a-e5f9-4256-8de4-3d9875a52945-kube-api-access-9qxc5\") pod \"managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5\" (UID: \"8e7cc21a-e5f9-4256-8de4-3d9875a52945\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"
Sep 07 10:12:38.253657 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-registry-certificates\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.253657 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqc87\" (UniqueName: \"kubernetes.io/projected/a81c91f6-0aed-48a4-b4c7-15acb3869234-kube-api-access-qqc87\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.253788 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-ca-trust-extracted\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.253788 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253702    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2h2c\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-kube-api-access-j2h2c\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.253878 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-image-registry-private-configuration\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.253878 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-registry-tls\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.253945 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a81c91f6-0aed-48a4-b4c7-15acb3869234-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.253945 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e7cc21a-e5f9-4256-8de4-3d9875a52945-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5\" (UID: \"8e7cc21a-e5f9-4256-8de4-3d9875a52945\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"
Sep 07 10:12:38.253945 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a81c91f6-0aed-48a4-b4c7-15acb3869234-tmp\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.254093 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253949    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-trusted-ca\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.254093 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-installation-pull-secrets\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.254093 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.253980    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-bound-sa-token\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.254347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.254327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a81c91f6-0aed-48a4-b4c7-15acb3869234-tmp\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.256253 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.256231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a81c91f6-0aed-48a4-b4c7-15acb3869234-klusterlet-config\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.256349 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.256270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e7cc21a-e5f9-4256-8de4-3d9875a52945-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5\" (UID: \"8e7cc21a-e5f9-4256-8de4-3d9875a52945\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"
Sep 07 10:12:38.261506 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.261484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxc5\" (UniqueName: \"kubernetes.io/projected/8e7cc21a-e5f9-4256-8de4-3d9875a52945-kube-api-access-9qxc5\") pod \"managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5\" (UID: \"8e7cc21a-e5f9-4256-8de4-3d9875a52945\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"
Sep 07 10:12:38.261646 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.261629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqc87\" (UniqueName: \"kubernetes.io/projected/a81c91f6-0aed-48a4-b4c7-15acb3869234-kube-api-access-qqc87\") pod \"klusterlet-addon-workmgr-6dcc48db84-8xm6k\" (UID: \"a81c91f6-0aed-48a4-b4c7-15acb3869234\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.285270 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.285249    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:38.290393 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.290366    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:38.295912 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.295894    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:38.354430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-trusted-ca\") pod \"889199f9-100b-4bb5-86a1-66584c032c22\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") "
Sep 07 10:12:38.354430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354411    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-bound-sa-token\") pod \"889199f9-100b-4bb5-86a1-66584c032c22\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") "
Sep 07 10:12:38.354604 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354434    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgn84\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-kube-api-access-sgn84\") pod \"889199f9-100b-4bb5-86a1-66584c032c22\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") "
Sep 07 10:12:38.354604 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354457    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-registry-certificates\") pod \"889199f9-100b-4bb5-86a1-66584c032c22\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") "
Sep 07 10:12:38.354604 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354480    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-image-registry-private-configuration\") pod \"889199f9-100b-4bb5-86a1-66584c032c22\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") "
Sep 07 10:12:38.354604 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354505    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/889199f9-100b-4bb5-86a1-66584c032c22-ca-trust-extracted\") pod \"889199f9-100b-4bb5-86a1-66584c032c22\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") "
Sep 07 10:12:38.354604 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354550    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-installation-pull-secrets\") pod \"889199f9-100b-4bb5-86a1-66584c032c22\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") "
Sep 07 10:12:38.354854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-ca-trust-extracted\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.354854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2h2c\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-kube-api-access-j2h2c\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.354854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-image-registry-private-configuration\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.354854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354723    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-registry-tls\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.354854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354808    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "889199f9-100b-4bb5-86a1-66584c032c22" (UID: "889199f9-100b-4bb5-86a1-66584c032c22"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:12:38.354854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354816    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "889199f9-100b-4bb5-86a1-66584c032c22" (UID: "889199f9-100b-4bb5-86a1-66584c032c22"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:12:38.354854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-trusted-ca\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.355204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-installation-pull-secrets\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.355204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-bound-sa-token\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.355204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-registry-certificates\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.355204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.354993    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-trusted-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:12:38.355204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.355011    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/889199f9-100b-4bb5-86a1-66584c032c22-registry-certificates\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:12:38.355204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.355089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-ca-trust-extracted\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.355494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.355325    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/889199f9-100b-4bb5-86a1-66584c032c22-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "889199f9-100b-4bb5-86a1-66584c032c22" (UID: "889199f9-100b-4bb5-86a1-66584c032c22"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:12:38.355918 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.355895    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-trusted-ca\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.356087 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.356002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-registry-certificates\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.357202 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.357156    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "889199f9-100b-4bb5-86a1-66584c032c22" (UID: "889199f9-100b-4bb5-86a1-66584c032c22"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:12:38.357425 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.357406    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "889199f9-100b-4bb5-86a1-66584c032c22" (UID: "889199f9-100b-4bb5-86a1-66584c032c22"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:12:38.357558 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.357536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-kube-api-access-sgn84" (OuterVolumeSpecName: "kube-api-access-sgn84") pod "889199f9-100b-4bb5-86a1-66584c032c22" (UID: "889199f9-100b-4bb5-86a1-66584c032c22"). InnerVolumeSpecName "kube-api-access-sgn84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:12:38.357624 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.357566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-image-registry-private-configuration\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.357808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.357789    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "889199f9-100b-4bb5-86a1-66584c032c22" (UID: "889199f9-100b-4bb5-86a1-66584c032c22"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:12:38.358270 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.358247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-installation-pull-secrets\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.358651 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.358628    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-registry-tls\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.373950 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.373930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2h2c\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-kube-api-access-j2h2c\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.378306 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.378290    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b-bound-sa-token\") pod \"image-registry-6587d68c88-wz6c8\" (UID: \"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b\") " pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.390236 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.390212    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:38.403953 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.403929    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:38.408684 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.408669    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"
Sep 07 10:12:38.455897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.455828    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgn84\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-kube-api-access-sgn84\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:12:38.455897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.455852    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-image-registry-private-configuration\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:12:38.455897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.455861    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/889199f9-100b-4bb5-86a1-66584c032c22-ca-trust-extracted\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:12:38.455897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.455871    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/889199f9-100b-4bb5-86a1-66584c032c22-installation-pull-secrets\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:12:38.455897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.455880    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-bound-sa-token\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:12:38.498543 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.498516    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6hpk4\""
Sep 07 10:12:38.501674 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.501650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:38.507453 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.507436    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:38.602397 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.602372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:38.663854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.663790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:38.663854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.663826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:38.664007 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.663878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:38.666548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.666360    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 10:12:38.666693 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.666619    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 10:12:38.666693 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.666645    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 10:12:38.667194 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.667174    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"]
Sep 07 10:12:38.670134 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:38.670110    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81c91f6_0aed_48a4_b4c7_15acb3869234.slice/crio-c9a75ef2deee4f55796331c93dfce8039b1dc6f83d5e5f3d808e2ca74f2e2a73 WatchSource:0}: Error finding container c9a75ef2deee4f55796331c93dfce8039b1dc6f83d5e5f3d808e2ca74f2e2a73: Status 404 returned error can't find the container with id c9a75ef2deee4f55796331c93dfce8039b1dc6f83d5e5f3d808e2ca74f2e2a73
Sep 07 10:12:38.676346 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.676327    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 10:12:38.676565 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.676547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bfd233c5-b9bd-47b1-aee1-1013a76983b5-original-pull-secret\") pod \"global-pull-secret-syncer-jr9hg\" (UID: \"bfd233c5-b9bd-47b1-aee1-1013a76983b5\") " pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:38.676624 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.676592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/232a763f-6ba8-4484-855b-ead84890b628-metrics-certs\") pod \"network-metrics-daemon-dbjml\" (UID: \"232a763f-6ba8-4484-855b-ead84890b628\") " pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:38.686639 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.686621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8szd\" (UniqueName: \"kubernetes.io/projected/98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d-kube-api-access-s8szd\") pod \"network-check-target-zzfpd\" (UID: \"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d\") " pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:38.688136 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.688102    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5"]
Sep 07 10:12:38.691785 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:38.691765    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e7cc21a_e5f9_4256_8de4_3d9875a52945.slice/crio-50f34c30241f35eec6ce7f2d0e3c9b5bdfc6683b34b12ad1125738558ed0c89f WatchSource:0}: Error finding container 50f34c30241f35eec6ce7f2d0e3c9b5bdfc6683b34b12ad1125738558ed0c89f: Status 404 returned error can't find the container with id 50f34c30241f35eec6ce7f2d0e3c9b5bdfc6683b34b12ad1125738558ed0c89f
Sep 07 10:12:38.787876 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.787847    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6587d68c88-wz6c8"]
Sep 07 10:12:38.790248 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:38.790226    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6e72ba_6fa2_44ab_8d5f_8b943f196c1b.slice/crio-97f63dda0589a386ef5afdd984b40fa55b9dbb4a66ece43caf4118e16003e298 WatchSource:0}: Error finding container 97f63dda0589a386ef5afdd984b40fa55b9dbb4a66ece43caf4118e16003e298: Status 404 returned error can't find the container with id 97f63dda0589a386ef5afdd984b40fa55b9dbb4a66ece43caf4118e16003e298
Sep 07 10:12:38.901372 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.901349    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9hg"
Sep 07 10:12:38.909416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.909396    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v9xn8\""
Sep 07 10:12:38.914679 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.914640    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cp5xp\""
Sep 07 10:12:38.917530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.917513    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:38.922688 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:38.922669    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dbjml"
Sep 07 10:12:39.167892 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.167820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:39.168269 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.167988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:39.170208 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.170185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4ab0827d-36bd-4e44-b152-b72cf5cf8861-metrics-tls\") pod \"dns-default-nj8x8\" (UID: \"4ab0827d-36bd-4e44-b152-b72cf5cf8861\") " pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:39.170322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.170233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"image-registry-58c774d8d7-m4f7f\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") " pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:39.182163 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.182137    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jr9hg"]
Sep 07 10:12:39.190978 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:39.190952    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd233c5_b9bd_47b1_aee1_1013a76983b5.slice/crio-ebaa68516d830cfe263c280b72e3515fadf8218b5dc7b7fd42ac5fa46b7fd92a WatchSource:0}: Error finding container ebaa68516d830cfe263c280b72e3515fadf8218b5dc7b7fd42ac5fa46b7fd92a: Status 404 returned error can't find the container with id ebaa68516d830cfe263c280b72e3515fadf8218b5dc7b7fd42ac5fa46b7fd92a
Sep 07 10:12:39.203502 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.203477    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zzfpd"]
Sep 07 10:12:39.207317 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:39.207294    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cc33d2_7ac4_4c2e_b6cd_aa985d451c5d.slice/crio-08f045d21bbafb6642aeea3c273d36968264e0bbc5bdd7c8b2390d8982f9ba7b WatchSource:0}: Error finding container 08f045d21bbafb6642aeea3c273d36968264e0bbc5bdd7c8b2390d8982f9ba7b: Status 404 returned error can't find the container with id 08f045d21bbafb6642aeea3c273d36968264e0bbc5bdd7c8b2390d8982f9ba7b
Sep 07 10:12:39.227428 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.227408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dbjml"]
Sep 07 10:12:39.229780 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:39.229760    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232a763f_6ba8_4484_855b_ead84890b628.slice/crio-9e00febf9836c1cf94c3b2b79ad649df34065df38499476254ac39680f245d87 WatchSource:0}: Error finding container 9e00febf9836c1cf94c3b2b79ad649df34065df38499476254ac39680f245d87: Status 404 returned error can't find the container with id 9e00febf9836c1cf94c3b2b79ad649df34065df38499476254ac39680f245d87
Sep 07 10:12:39.268668 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.268640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") pod \"889199f9-100b-4bb5-86a1-66584c032c22\" (UID: \"889199f9-100b-4bb5-86a1-66584c032c22\") "
Sep 07 10:12:39.270709 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.270684    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "889199f9-100b-4bb5-86a1-66584c032c22" (UID: "889199f9-100b-4bb5-86a1-66584c032c22"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:12:39.289125 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.289100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5" event={"ID":"8e7cc21a-e5f9-4256-8de4-3d9875a52945","Type":"ContainerStarted","Data":"50f34c30241f35eec6ce7f2d0e3c9b5bdfc6683b34b12ad1125738558ed0c89f"}
Sep 07 10:12:39.290355 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.290329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zzfpd" event={"ID":"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d","Type":"ContainerStarted","Data":"08f045d21bbafb6642aeea3c273d36968264e0bbc5bdd7c8b2390d8982f9ba7b"}
Sep 07 10:12:39.291535 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.291512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbjml" event={"ID":"232a763f-6ba8-4484-855b-ead84890b628","Type":"ContainerStarted","Data":"9e00febf9836c1cf94c3b2b79ad649df34065df38499476254ac39680f245d87"}
Sep 07 10:12:39.292796 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.292734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jr9hg" event={"ID":"bfd233c5-b9bd-47b1-aee1-1013a76983b5","Type":"ContainerStarted","Data":"ebaa68516d830cfe263c280b72e3515fadf8218b5dc7b7fd42ac5fa46b7fd92a"}
Sep 07 10:12:39.294385 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.294361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6587d68c88-wz6c8" event={"ID":"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b","Type":"ContainerStarted","Data":"235a48d680806b297a95a8b1fa3aade3d3778d4646ba91e7dca367d9bf89ad8c"}
Sep 07 10:12:39.294502 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.294397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6587d68c88-wz6c8" event={"ID":"3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b","Type":"ContainerStarted","Data":"97f63dda0589a386ef5afdd984b40fa55b9dbb4a66ece43caf4118e16003e298"}
Sep 07 10:12:39.294502 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.294424    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:12:39.295532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.295509    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k" event={"ID":"a81c91f6-0aed-48a4-b4c7-15acb3869234","Type":"ContainerStarted","Data":"c9a75ef2deee4f55796331c93dfce8039b1dc6f83d5e5f3d808e2ca74f2e2a73"}
Sep 07 10:12:39.295532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.295523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58c774d8d7-m4f7f"
Sep 07 10:12:39.317711 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.315399    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6587d68c88-wz6c8" podStartSLOduration=1.315381671 podStartE2EDuration="1.315381671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:39.313277294 +0000 UTC m=+65.861792272" watchObservedRunningTime="2026-09-07 10:12:39.315381671 +0000 UTC m=+65.863896659"
Sep 07 10:12:39.340892 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.340866    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58c774d8d7-m4f7f"]
Sep 07 10:12:39.344851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.344831    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58c774d8d7-m4f7f"]
Sep 07 10:12:39.369461 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.369438    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/889199f9-100b-4bb5-86a1-66584c032c22-registry-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:12:39.429383 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.429334    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6d8d2\""
Sep 07 10:12:39.438159 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.438137    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:39.733116 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.733026    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nj8x8"]
Sep 07 10:12:39.739145 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:39.739109    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab0827d_36bd_4e44_b152_b72cf5cf8861.slice/crio-a16f4ff8a83d2514593c41ec4563f0c3a95157cc10c9e561fd41cc7d11818b2d WatchSource:0}: Error finding container a16f4ff8a83d2514593c41ec4563f0c3a95157cc10c9e561fd41cc7d11818b2d: Status 404 returned error can't find the container with id a16f4ff8a83d2514593c41ec4563f0c3a95157cc10c9e561fd41cc7d11818b2d
Sep 07 10:12:39.842761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.842698    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:39.898147 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.897915    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="889199f9-100b-4bb5-86a1-66584c032c22" path="/var/lib/kubelet/pods/889199f9-100b-4bb5-86a1-66584c032c22/volumes"
Sep 07 10:12:39.942214 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:39.942093    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:40.008581 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.007538    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"]
Sep 07 10:12:40.029392 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.029348    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:40.031148 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.031122    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"]
Sep 07 10:12:40.031264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.031230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"
Sep 07 10:12:40.035576 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.035001    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 07 10:12:40.035576 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.035252    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xk5tg\""
Sep 07 10:12:40.074739 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.074600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8f063994-2fa4-4b52-a4bf-3730feb3f383-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8clrt\" (UID: \"8f063994-2fa4-4b52-a4bf-3730feb3f383\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"
Sep 07 10:12:40.157345 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.157217    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:40.175444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.175011    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8f063994-2fa4-4b52-a4bf-3730feb3f383-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8clrt\" (UID: \"8f063994-2fa4-4b52-a4bf-3730feb3f383\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"
Sep 07 10:12:40.175444 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:40.175157    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 07 10:12:40.175444 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:12:40.175232    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f063994-2fa4-4b52-a4bf-3730feb3f383-tls-certificates podName:8f063994-2fa4-4b52-a4bf-3730feb3f383 nodeName:}" failed. No retries permitted until 2026-09-07 10:12:40.675211567 +0000 UTC m=+67.223726561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/8f063994-2fa4-4b52-a4bf-3730feb3f383-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-8clrt" (UID: "8f063994-2fa4-4b52-a4bf-3730feb3f383") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 07 10:12:40.260760 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.260583    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:40.305147 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.305118    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nj8x8" event={"ID":"4ab0827d-36bd-4e44-b152-b72cf5cf8861","Type":"ContainerStarted","Data":"a16f4ff8a83d2514593c41ec4563f0c3a95157cc10c9e561fd41cc7d11818b2d"}
Sep 07 10:12:40.357057 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.357024    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:40.680345 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.680306    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8f063994-2fa4-4b52-a4bf-3730feb3f383-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8clrt\" (UID: \"8f063994-2fa4-4b52-a4bf-3730feb3f383\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"
Sep 07 10:12:40.690743 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.690678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/8f063994-2fa4-4b52-a4bf-3730feb3f383-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8clrt\" (UID: \"8f063994-2fa4-4b52-a4bf-3730feb3f383\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"
Sep 07 10:12:40.946259 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:40.945922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"
Sep 07 10:12:41.434324 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.433292    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b4b557ff8-m6rm2"]
Sep 07 10:12:41.465602 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.465552    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4b557ff8-m6rm2"]
Sep 07 10:12:41.465763 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.465684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.468615 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.468590    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 10:12:41.468996 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.468976    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 10:12:41.469233 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.469216    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 10:12:41.469447 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.469430    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ndkq9\""
Sep 07 10:12:41.469764 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.469746    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 10:12:41.469959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.469944    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 10:12:41.470207 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.470190    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 10:12:41.470409 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.470393    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 10:12:41.588859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.588821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swwct\" (UniqueName: \"kubernetes.io/projected/c853d585-23d4-4a0b-a96e-2dea59a9baec-kube-api-access-swwct\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.589018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.588873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-serving-cert\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.589018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.588901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-config\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.589018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.588930    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-service-ca\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.589018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.588969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-oauth-serving-cert\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.589264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.589034    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-oauth-config\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.690030 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.689954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-oauth-config\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.690229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.690027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swwct\" (UniqueName: \"kubernetes.io/projected/c853d585-23d4-4a0b-a96e-2dea59a9baec-kube-api-access-swwct\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.690229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.690057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-serving-cert\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.690229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.690099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-config\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.690229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.690126    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-service-ca\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.690229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.690164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-oauth-serving-cert\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.691535 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.691508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-config\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.691649 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.691524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-oauth-serving-cert\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.692305 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.692281    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-service-ca\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.696562 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.696508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-serving-cert\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.697156 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.697128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-oauth-config\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.699923 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.699896    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swwct\" (UniqueName: \"kubernetes.io/projected/c853d585-23d4-4a0b-a96e-2dea59a9baec-kube-api-access-swwct\") pod \"console-b4b557ff8-m6rm2\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") " pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:41.783908 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:41.783720    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:44.136427 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:44.136346    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"]
Sep 07 10:12:45.421815 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:45.421778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:45.424556 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:45.424506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/89b831f6-f4b9-4ddf-baed-806b217f0c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6ft8s\" (UID: \"89b831f6-f4b9-4ddf-baed-806b217f0c8e\") " pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:45.601421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:45.601388    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gp5c5\""
Sep 07 10:12:45.609243 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:45.609219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6ft8s"
Sep 07 10:12:46.138620 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:46.138580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:46.194764 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:46.194729    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:46.402532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:46.402446    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:46.454722 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:46.454690    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:46.808960 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:46.808930    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:46.928961 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:46.928928    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:47.211720 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:47.211614    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:47.268493 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:47.268449    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:47.307351 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:47.307326    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f063994_2fa4_4b52_a4bf_3730feb3f383.slice/crio-34e8c909fbea94f970483ca65ad2e5233a6b3e9c448e559ea15233b486b28420 WatchSource:0}: Error finding container 34e8c909fbea94f970483ca65ad2e5233a6b3e9c448e559ea15233b486b28420: Status 404 returned error can't find the container with id 34e8c909fbea94f970483ca65ad2e5233a6b3e9c448e559ea15233b486b28420
Sep 07 10:12:47.326232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:47.326208    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt" event={"ID":"8f063994-2fa4-4b52-a4bf-3730feb3f383","Type":"ContainerStarted","Data":"34e8c909fbea94f970483ca65ad2e5233a6b3e9c448e559ea15233b486b28420"}
Sep 07 10:12:47.671087 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:47.671040    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4b557ff8-m6rm2"]
Sep 07 10:12:47.675940 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:47.675909    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc853d585_23d4_4a0b_a96e_2dea59a9baec.slice/crio-a0cb93056027175a873f7d682f15b559a8e805bbca3c65e0c78818e56eda557f WatchSource:0}: Error finding container a0cb93056027175a873f7d682f15b559a8e805bbca3c65e0c78818e56eda557f: Status 404 returned error can't find the container with id a0cb93056027175a873f7d682f15b559a8e805bbca3c65e0c78818e56eda557f
Sep 07 10:12:47.695662 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:47.695633    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6ft8s"]
Sep 07 10:12:47.699102 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:47.699059    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b831f6_f4b9_4ddf_baed_806b217f0c8e.slice/crio-b2e90aa8d1e20e53badf183c35faf864a38dcf3a34a75b3bd5f5f191a5d7a3e0 WatchSource:0}: Error finding container b2e90aa8d1e20e53badf183c35faf864a38dcf3a34a75b3bd5f5f191a5d7a3e0: Status 404 returned error can't find the container with id b2e90aa8d1e20e53badf183c35faf864a38dcf3a34a75b3bd5f5f191a5d7a3e0
Sep 07 10:12:48.335137 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.335097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k" event={"ID":"a81c91f6-0aed-48a4-b4c7-15acb3869234","Type":"ContainerStarted","Data":"735186a5fcc2db8ff8f88a78bf43b2451b0a879d517786162de063f84e7c301d"}
Sep 07 10:12:48.335427 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.335373    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:48.337410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.337382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5" event={"ID":"8e7cc21a-e5f9-4256-8de4-3d9875a52945","Type":"ContainerStarted","Data":"3a377b4d3825ee4ea4b1f55acd1eabb4edb15cbfa01b2c328d02fa8a10faea3a"}
Sep 07 10:12:48.337667 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.337642    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k"
Sep 07 10:12:48.339793 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.339769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zzfpd" event={"ID":"98cc33d2-7ac4-4c2e-b6cd-aa985d451c5d","Type":"ContainerStarted","Data":"98897669d8d8c216d6bc6d283d45257c5b313eaf1e083b62953fd78adda40bb5"}
Sep 07 10:12:48.339928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.339913    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:12:48.342505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.342375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nj8x8" event={"ID":"4ab0827d-36bd-4e44-b152-b72cf5cf8861","Type":"ContainerStarted","Data":"d406322b9e85607ba997209ff1f148bf67e322054580d7168f6491e0e60e54dd"}
Sep 07 10:12:48.342505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.342419    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nj8x8" event={"ID":"4ab0827d-36bd-4e44-b152-b72cf5cf8861","Type":"ContainerStarted","Data":"8c6bdc31ea92378e96e5dd92bd301a2cd59ef88583d9426dc0a8d9e30b2b1e23"}
Sep 07 10:12:48.342505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.342470    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:48.344127 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.344093    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ft8s" event={"ID":"89b831f6-f4b9-4ddf-baed-806b217f0c8e","Type":"ContainerStarted","Data":"0ce50b58e5917f07830b3ba39b91f22dc445cdf084266736a3cc1c2835ae0d56"}
Sep 07 10:12:48.344208 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.344144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ft8s" event={"ID":"89b831f6-f4b9-4ddf-baed-806b217f0c8e","Type":"ContainerStarted","Data":"b2e90aa8d1e20e53badf183c35faf864a38dcf3a34a75b3bd5f5f191a5d7a3e0"}
Sep 07 10:12:48.345731 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.345704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b557ff8-m6rm2" event={"ID":"c853d585-23d4-4a0b-a96e-2dea59a9baec","Type":"ContainerStarted","Data":"a0cb93056027175a873f7d682f15b559a8e805bbca3c65e0c78818e56eda557f"}
Sep 07 10:12:48.349892 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.349871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbjml" event={"ID":"232a763f-6ba8-4484-855b-ead84890b628","Type":"ContainerStarted","Data":"f7e90cc6e841a729f2ea312daf18feef8460402fccae623c9cc82553b78f6ffb"}
Sep 07 10:12:48.349990 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.349913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dbjml" event={"ID":"232a763f-6ba8-4484-855b-ead84890b628","Type":"ContainerStarted","Data":"3d4cda318c3ce1ffd93ae15008324a6796faaa435270bdcf1543dfdf7091327f"}
Sep 07 10:12:48.351942 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.351910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jr9hg" event={"ID":"bfd233c5-b9bd-47b1-aee1-1013a76983b5","Type":"ContainerStarted","Data":"134cd123135e148107b1befd114f8b3ab11063ec5373adbeb510f5a7e63412bc"}
Sep 07 10:12:48.353633 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.353590    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6dcc48db84-8xm6k" podStartSLOduration=1.607217566 podStartE2EDuration="10.353575546s" totalImagesPullingTime="8.74635798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:48.352863825 +0000 UTC m=+74.901378825" watchObservedRunningTime="2026-09-07 10:12:48.353575546 +0000 UTC m=+74.902090534"
Sep 07 10:12:48.368840 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.368440    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dbjml" podStartSLOduration=66.216806446 podStartE2EDuration="1m14.368420866s" totalImagesPullingTime="8.15161442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:48.367046153 +0000 UTC m=+74.915561141" watchObservedRunningTime="2026-09-07 10:12:48.368420866 +0000 UTC m=+74.916935856"
Sep 07 10:12:48.386023 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.385580    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f68b8dd56-6dpv5" podStartSLOduration=1.692199325 podStartE2EDuration="10.385563754s" totalImagesPullingTime="8.693364429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:48.38479684 +0000 UTC m=+74.933311828" watchObservedRunningTime="2026-09-07 10:12:48.385563754 +0000 UTC m=+74.934078742"
Sep 07 10:12:48.401360 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.400689    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jr9hg" podStartSLOduration=66.182301502 podStartE2EDuration="1m14.400674955s" totalImagesPullingTime="8.218373453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:48.399369923 +0000 UTC m=+74.947884911" watchObservedRunningTime="2026-09-07 10:12:48.400674955 +0000 UTC m=+74.949189938"
Sep 07 10:12:48.416098 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.416030    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zzfpd" podStartSLOduration=67.221895611 podStartE2EDuration="1m15.416013088s" totalImagesPullingTime="8.194117477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:11:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:48.415450035 +0000 UTC m=+74.963965023" watchObservedRunningTime="2026-09-07 10:12:48.416013088 +0000 UTC m=+74.964528075"
Sep 07 10:12:48.448346 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.448297    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nj8x8" podStartSLOduration=33.79581119 podStartE2EDuration="41.448282247s" totalImagesPullingTime="7.652471057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:48.446967191 +0000 UTC m=+74.995482179" watchObservedRunningTime="2026-09-07 10:12:48.448282247 +0000 UTC m=+74.996797234"
Sep 07 10:12:48.467740 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.467703    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:48.526806 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:48.526773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:49.130733 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.130697    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:49.185677 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.185637    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:49.362448 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.362411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ft8s" event={"ID":"89b831f6-f4b9-4ddf-baed-806b217f0c8e","Type":"ContainerStarted","Data":"7a0b88ae9cea17da603dcba03940e21794a243f9a1673f3aba6c1b202deb14e1"}
Sep 07 10:12:49.364869 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.364394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt" event={"ID":"8f063994-2fa4-4b52-a4bf-3730feb3f383","Type":"ContainerStarted","Data":"a34ee0cc026e6855521a8f9eb93ff65c249e1902e5d328bb927b677d511ee8a7"}
Sep 07 10:12:49.380356 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.380311    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt" podStartSLOduration=8.960252354 podStartE2EDuration="10.380296986s" totalImagesPullingTime="1.420044632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:49.379686986 +0000 UTC m=+75.928201974" watchObservedRunningTime="2026-09-07 10:12:49.380296986 +0000 UTC m=+75.928812017"
Sep 07 10:12:49.750524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.750490    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b85794bc8-48l96"]
Sep 07 10:12:49.753488 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.753468    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.762556 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.762525    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 10:12:49.764980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.764958    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b85794bc8-48l96"]
Sep 07 10:12:49.858283 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.858245    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-oauth-config\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.858283 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.858284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-trusted-ca-bundle\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.858521 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.858329    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-serving-cert\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.858521 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.858408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-config\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.858521 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.858454    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-service-ca\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.858521 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.858488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-oauth-serving-cert\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.858674 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.858522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgxx\" (UniqueName: \"kubernetes.io/projected/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-kube-api-access-ldgxx\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.959757 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.959718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-serving-cert\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.959930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.959780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-config\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.959930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.959816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-service-ca\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.959930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.959841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-oauth-serving-cert\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.959930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.959874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgxx\" (UniqueName: \"kubernetes.io/projected/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-kube-api-access-ldgxx\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.959930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.959916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-oauth-config\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.960204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.960146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-trusted-ca-bundle\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.960664 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.960617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-service-ca\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.960745 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.960680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-config\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.960745 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.960694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-oauth-serving-cert\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.961089 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.961054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-trusted-ca-bundle\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.962636 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.962616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-oauth-config\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.962721 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.962693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-serving-cert\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:49.968114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:49.968049    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgxx\" (UniqueName: \"kubernetes.io/projected/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-kube-api-access-ldgxx\") pod \"console-7b85794bc8-48l96\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") " pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:50.063280 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:50.063237    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:12:50.368984 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:50.368915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"
Sep 07 10:12:50.375212 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:50.375188    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8clrt"
Sep 07 10:12:51.040939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.040909    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:51.110369 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.110174    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:51.321100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.321015    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b85794bc8-48l96"]
Sep 07 10:12:51.323799 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:51.323775    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec24e20b_60aa_43c0_8c11_0ff9b1e5fef3.slice/crio-6a072a2dd25b630e7b0d357c5bf7c52649e3390357701f6d244d55e6a2f396e1 WatchSource:0}: Error finding container 6a072a2dd25b630e7b0d357c5bf7c52649e3390357701f6d244d55e6a2f396e1: Status 404 returned error can't find the container with id 6a072a2dd25b630e7b0d357c5bf7c52649e3390357701f6d244d55e6a2f396e1
Sep 07 10:12:51.373255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.373227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b85794bc8-48l96" event={"ID":"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3","Type":"ContainerStarted","Data":"6a072a2dd25b630e7b0d357c5bf7c52649e3390357701f6d244d55e6a2f396e1"}
Sep 07 10:12:51.375323 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.375286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6ft8s" event={"ID":"89b831f6-f4b9-4ddf-baed-806b217f0c8e","Type":"ContainerStarted","Data":"77742cc49c49c78297f57d5bab77f9f059824a0a850030361fa48ad89ac88d67"}
Sep 07 10:12:51.376975 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.376953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b557ff8-m6rm2" event={"ID":"c853d585-23d4-4a0b-a96e-2dea59a9baec","Type":"ContainerStarted","Data":"ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2"}
Sep 07 10:12:51.394119 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.394046    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6ft8s" podStartSLOduration=35.18707648 podStartE2EDuration="38.394028002s" totalImagesPullingTime="3.206951522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:12:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:51.392750872 +0000 UTC m=+77.941265860" watchObservedRunningTime="2026-09-07 10:12:51.394028002 +0000 UTC m=+77.942542991"
Sep 07 10:12:51.398091 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.395268    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:51.414967 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.414913    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b4b557ff8-m6rm2" podStartSLOduration=7.035549489 podStartE2EDuration="10.414899671s" totalImagesPullingTime="3.379350182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:51.414003584 +0000 UTC m=+77.962518573" watchObservedRunningTime="2026-09-07 10:12:51.414899671 +0000 UTC m=+77.963414659"
Sep 07 10:12:51.729446 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.729347    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:51.784448 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.784419    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:51.784554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.784497    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:51.788974 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.788950    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:51.809231 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:51.809207    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:52.382438 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:52.382397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b85794bc8-48l96" event={"ID":"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3","Type":"ContainerStarted","Data":"113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1"}
Sep 07 10:12:52.386341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:52.386318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:12:52.402626 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:52.402588    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b85794bc8-48l96" podStartSLOduration=3.402574128 podStartE2EDuration="3.402574128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:52.402128006 +0000 UTC m=+78.950643009" watchObservedRunningTime="2026-09-07 10:12:52.402574128 +0000 UTC m=+78.951089114"
Sep 07 10:12:53.868906 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:53.868854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:55.888811 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:55.888773    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:55.989581 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:55.989542    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.192769 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.192683    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.301848 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.301725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.402343 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.402308    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.485010 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.484938    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"]
Sep 07 10:12:56.488159 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.488139    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.490719 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.490694    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 07 10:12:56.491167 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.491147    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 10:12:56.491167 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.491162    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 07 10:12:56.491323 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.491233    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 10:12:56.492102 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.492084    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zw9jp\""
Sep 07 10:12:56.492188 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.492085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 10:12:56.498043 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.498018    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.498683 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.498658    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"]
Sep 07 10:12:56.517801 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.517775    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-px2dm"]
Sep 07 10:12:56.520625 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.520607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.522728 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.522703    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 10:12:56.522938 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.522924    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 10:12:56.523019 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.523005    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s5st7\""
Sep 07 10:12:56.523138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.523123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 10:12:56.583775 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.583749    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.611365 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-textfile\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.611365 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.611580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-tls\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.611580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.611580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611422    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-wtmp\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.611580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611496    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.611580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46z2n\" (UniqueName: \"kubernetes.io/projected/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-kube-api-access-46z2n\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.611580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611555    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.611799 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-root\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.611799 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611619    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83a941cf-17ea-4e2e-890d-d898be6b5d0b-metrics-client-ca\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.611799 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44rk7\" (UniqueName: \"kubernetes.io/projected/83a941cf-17ea-4e2e-890d-d898be6b5d0b-kube-api-access-44rk7\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.611799 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.611799 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.611725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-sys\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.681676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.681630    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.712893 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.712862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-sys\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713036 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.712922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-textfile\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713036 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.712948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713036 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.712969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-sys\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713036 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.712973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-tls\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713036 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.713289 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-wtmp\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713289 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713108    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.713289 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713224    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46z2n\" (UniqueName: \"kubernetes.io/projected/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-kube-api-access-46z2n\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.713289 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-wtmp\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713289 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.713533 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713298    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-root\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713533 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83a941cf-17ea-4e2e-890d-d898be6b5d0b-metrics-client-ca\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713533 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713355    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-textfile\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713533 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/83a941cf-17ea-4e2e-890d-d898be6b5d0b-root\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713533 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44rk7\" (UniqueName: \"kubernetes.io/projected/83a941cf-17ea-4e2e-890d-d898be6b5d0b-kube-api-access-44rk7\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.713533 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.713501    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.714026 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.714004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83a941cf-17ea-4e2e-890d-d898be6b5d0b-metrics-client-ca\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.714151 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.714037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.714199 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.714188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.715963 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.715940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.716082 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.716049    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.716134 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.716117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/83a941cf-17ea-4e2e-890d-d898be6b5d0b-node-exporter-tls\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.716169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.716143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.720541 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.720521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44rk7\" (UniqueName: \"kubernetes.io/projected/83a941cf-17ea-4e2e-890d-d898be6b5d0b-kube-api-access-44rk7\") pod \"node-exporter-px2dm\" (UID: \"83a941cf-17ea-4e2e-890d-d898be6b5d0b\") " pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.720761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.720744    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46z2n\" (UniqueName: \"kubernetes.io/projected/790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea-kube-api-access-46z2n\") pod \"openshift-state-metrics-657fd6c54-g754t\" (UID: \"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.781221 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.781104    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.797959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.797927    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"
Sep 07 10:12:56.829295 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.829253    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-px2dm"
Sep 07 10:12:56.841431 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:56.841405    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a941cf_17ea_4e2e_890d_d898be6b5d0b.slice/crio-fe5f88901e1876e18d63de8da6f209fff5b40930538a6df42864012a434d10e8 WatchSource:0}: Error finding container fe5f88901e1876e18d63de8da6f209fff5b40930538a6df42864012a434d10e8: Status 404 returned error can't find the container with id fe5f88901e1876e18d63de8da6f209fff5b40930538a6df42864012a434d10e8
Sep 07 10:12:56.881548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.881519    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:56.982579 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:56.982548    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:57.070090 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.070046    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-g754t"]
Sep 07 10:12:57.072923 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:57.072892    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790b4027_9f0b_4b46_9e4a_89c2ddf6f6ea.slice/crio-5a2050f693d79b8feb12349ab67e33e7c732922c7488853794b39fcddda9779d WatchSource:0}: Error finding container 5a2050f693d79b8feb12349ab67e33e7c732922c7488853794b39fcddda9779d: Status 404 returned error can't find the container with id 5a2050f693d79b8feb12349ab67e33e7c732922c7488853794b39fcddda9779d
Sep 07 10:12:57.077541 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.077516    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:57.146781 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.146753    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:57.400331 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.400238    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px2dm" event={"ID":"83a941cf-17ea-4e2e-890d-d898be6b5d0b","Type":"ContainerStarted","Data":"fe5f88901e1876e18d63de8da6f209fff5b40930538a6df42864012a434d10e8"}
Sep 07 10:12:57.402598 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.402571    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t" event={"ID":"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea","Type":"ContainerStarted","Data":"e969df8aeae8429cf85db1067fd087e1f0eec3d2bdc26097a38f1fe17dea9383"}
Sep 07 10:12:57.402735 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.402626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t" event={"ID":"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea","Type":"ContainerStarted","Data":"4c80d0739b32be37decde198cf725cf74dc26905e0a28ad505e691f5e99349ff"}
Sep 07 10:12:57.402735 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.402656    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t" event={"ID":"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea","Type":"ContainerStarted","Data":"5a2050f693d79b8feb12349ab67e33e7c732922c7488853794b39fcddda9779d"}
Sep 07 10:12:57.417347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.417313    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:57.504925 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.504884    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:57.510819 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.510780    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 10:12:57.514457 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.514427    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.517052 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517024    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 07 10:12:57.517203 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517092    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 07 10:12:57.517203 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517133    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 07 10:12:57.517203 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517028    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 07 10:12:57.517371 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517318    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 07 10:12:57.517485 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517469    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 07 10:12:57.517567 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517470    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 07 10:12:57.517620 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517568    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c5w5s\""
Sep 07 10:12:57.517862 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517692    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 07 10:12:57.517862 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.517726    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 07 10:12:57.528785 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.528763    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 10:12:57.617895 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.617857    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:57.619915 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.619889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5w66\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-kube-api-access-t5w66\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620054 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.619919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620054 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.619951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620054 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.619997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-out\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620168    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620358 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620358 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620437 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620371    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-web-config\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620437 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620510 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.620510 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.620479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.721668 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.721577    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.721668 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.721617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-out\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.721668 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.721639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.721935 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.721895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.721992 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.721936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.721992 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.721960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.722118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.721997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.722118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.722050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-web-config\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.722118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.722098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.722273 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.722139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.722273 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.722172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.722273 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.722202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5w66\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-kube-api-access-t5w66\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.722273 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.722229    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.724032 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.723036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.724032 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.723212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.724032 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.723987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.725709 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.725259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-out\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.725709 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.725671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.726147 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.726107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.726520 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.726450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.726609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.726561    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.726785 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.726764    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.727350 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.727324    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.727427 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.727416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-web-config\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.728367 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.728345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.730936 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.730913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5w66\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-kube-api-access-t5w66\") pod \"alertmanager-main-0\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:57.750751 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.750720    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:57.824980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:57.824957    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:12:58.109859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:58.109751    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 10:12:58.112799 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:12:58.112773    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9cb02c_8b85_464d_b012_0a8ae11346f3.slice/crio-28f887d0b09ae52c2ac9b7e301b4dc7526a01a4c73fcf1865c044a333c57fd50 WatchSource:0}: Error finding container 28f887d0b09ae52c2ac9b7e301b4dc7526a01a4c73fcf1865c044a333c57fd50: Status 404 returned error can't find the container with id 28f887d0b09ae52c2ac9b7e301b4dc7526a01a4c73fcf1865c044a333c57fd50
Sep 07 10:12:58.367910 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:58.367818    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nj8x8"
Sep 07 10:12:58.409032 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:58.408909    2560 generic.go:356] "Generic (PLEG): container finished" podID="83a941cf-17ea-4e2e-890d-d898be6b5d0b" containerID="aa079548083e96e766e14b8658cc9f9883ff3b96c6779c87fa7df9faa34bc25e" exitCode=0
Sep 07 10:12:58.409032 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:58.409015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px2dm" event={"ID":"83a941cf-17ea-4e2e-890d-d898be6b5d0b","Type":"ContainerDied","Data":"aa079548083e96e766e14b8658cc9f9883ff3b96c6779c87fa7df9faa34bc25e"}
Sep 07 10:12:58.411328 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:58.411294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerStarted","Data":"28f887d0b09ae52c2ac9b7e301b4dc7526a01a4c73fcf1865c044a333c57fd50"}
Sep 07 10:12:59.372034 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.371996    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:59.417670 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.417637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px2dm" event={"ID":"83a941cf-17ea-4e2e-890d-d898be6b5d0b","Type":"ContainerStarted","Data":"8a6c8948a1fc4d2e575c407d630bf2665e66949d2c109b137381d223d2d393b6"}
Sep 07 10:12:59.417852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.417721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-px2dm" event={"ID":"83a941cf-17ea-4e2e-890d-d898be6b5d0b","Type":"ContainerStarted","Data":"befb4e72ff8dc3881e0ab598aba8e7a2adfc4b3733022587c98ae5bb361aa6c5"}
Sep 07 10:12:59.417852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.417747    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-d856f4f5c-f98w6"]
Sep 07 10:12:59.421369 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.421333    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t" event={"ID":"790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea","Type":"ContainerStarted","Data":"3cc9100df2a68395a06b6b3f5c7c751b90271c11ba80719f23eb05632f0bb785"}
Sep 07 10:12:59.421485 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.421436    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.421686 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.421650    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerID="2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8" exitCode=0
Sep 07 10:12:59.421895 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.421744    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerDied","Data":"2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8"}
Sep 07 10:12:59.423776 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.423758    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 07 10:12:59.423874 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.423795    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 07 10:12:59.423874 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.423801    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 07 10:12:59.423998 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.423980    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 07 10:12:59.424249 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.424218    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l7vv8\""
Sep 07 10:12:59.424680 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.424656    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 07 10:12:59.424765 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.424707    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-foq84bo9l0c1j\""
Sep 07 10:12:59.433316 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.433265    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d856f4f5c-f98w6"]
Sep 07 10:12:59.440481 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.440433    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-px2dm" podStartSLOduration=2.409555286 podStartE2EDuration="3.440419379s" totalImagesPullingTime="964.096981ms" totalInitContainerRuntime="66.767112ms" isStatefulPod=true podCreationTimestamp="2026-09-07 10:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:59.440104982 +0000 UTC m=+85.988619964" watchObservedRunningTime="2026-09-07 10:12:59.440419379 +0000 UTC m=+85.988934365"
Sep 07 10:12:59.457367 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.457320    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-g754t" podStartSLOduration=2.160360342 podStartE2EDuration="3.457304421s" totalImagesPullingTime="1.296944079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:12:59.456957006 +0000 UTC m=+86.005471998" watchObservedRunningTime="2026-09-07 10:12:59.457304421 +0000 UTC m=+86.005819402"
Sep 07 10:12:59.461586 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.461561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:59.538298 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.538261    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/330f5d33-f755-4067-8b32-86fd7d7cbf24-metrics-client-ca\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.538470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.538443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.538612 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.538587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-tls\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.538750 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.538618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-grpc-tls\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.538994 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.538975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shs9l\" (UniqueName: \"kubernetes.io/projected/330f5d33-f755-4067-8b32-86fd7d7cbf24-kube-api-access-shs9l\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.539088 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.539050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.539329 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.539302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.539422 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.539348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.613307 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.613265    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:59.639715 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.639672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.639859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.639724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.639859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.639760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/330f5d33-f755-4067-8b32-86fd7d7cbf24-metrics-client-ca\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.639859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.639790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.639859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.639831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-tls\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.639859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.639854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-grpc-tls\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.640114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.639893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shs9l\" (UniqueName: \"kubernetes.io/projected/330f5d33-f755-4067-8b32-86fd7d7cbf24-kube-api-access-shs9l\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.640114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.639927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.640665 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.640635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/330f5d33-f755-4067-8b32-86fd7d7cbf24-metrics-client-ca\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.642428 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.642372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.642520 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.642399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.642706 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.642688    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.642753 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.642736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.642791 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.642751    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-thanos-querier-tls\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.642937 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.642923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/330f5d33-f755-4067-8b32-86fd7d7cbf24-secret-grpc-tls\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.647722 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.647699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shs9l\" (UniqueName: \"kubernetes.io/projected/330f5d33-f755-4067-8b32-86fd7d7cbf24-kube-api-access-shs9l\") pod \"thanos-querier-d856f4f5c-f98w6\" (UID: \"330f5d33-f755-4067-8b32-86fd7d7cbf24\") " pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:12:59.670840 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.670815    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:12:59.731048 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:12:59.730997    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:13:00.006962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.006938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-d856f4f5c-f98w6"]
Sep 07 10:13:00.009185 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:13:00.009158    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330f5d33_f755_4067_8b32_86fd7d7cbf24.slice/crio-ce79127414f9d0868c4e3ea63e999c746b98a5aa9636c3a7dc0843e144facb74 WatchSource:0}: Error finding container ce79127414f9d0868c4e3ea63e999c746b98a5aa9636c3a7dc0843e144facb74: Status 404 returned error can't find the container with id ce79127414f9d0868c4e3ea63e999c746b98a5aa9636c3a7dc0843e144facb74
Sep 07 10:13:00.063436 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.063410    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:13:00.063548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.063466    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:13:00.067975 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.067956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:13:00.310642 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.310603    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6587d68c88-wz6c8"
Sep 07 10:13:00.425302 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.425253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:00.427870 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.427843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6" event={"ID":"330f5d33-f755-4067-8b32-86fd7d7cbf24","Type":"ContainerStarted","Data":"ce79127414f9d0868c4e3ea63e999c746b98a5aa9636c3a7dc0843e144facb74"}
Sep 07 10:13:00.431750 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.431732    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:13:00.474245 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.474220    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4b557ff8-m6rm2"]
Sep 07 10:13:00.526042 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.526009    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:00.632014 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.631737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:00.726385 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.726290    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:00.830208 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:00.830163    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:01.214933 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:01.214904    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:01.283361 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:01.283332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:01.338623 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:01.338586    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:01.435410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:01.435379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerStarted","Data":"c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753"}
Sep 07 10:13:01.435700 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:01.435460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerStarted","Data":"a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091"}
Sep 07 10:13:01.435700 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:01.435503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerStarted","Data":"f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd"}
Sep 07 10:13:02.441816 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.441744    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6" event={"ID":"330f5d33-f755-4067-8b32-86fd7d7cbf24","Type":"ContainerStarted","Data":"dc3027740fe7d85defa612fd23c0bc672515176d012cbd96df4799c818f118ea"}
Sep 07 10:13:02.442228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.441818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6" event={"ID":"330f5d33-f755-4067-8b32-86fd7d7cbf24","Type":"ContainerStarted","Data":"a468da024ebc9b749078c641f89271ce2e76b9d38f8fc83f1367095f5f0da6ec"}
Sep 07 10:13:02.442228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.441852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6" event={"ID":"330f5d33-f755-4067-8b32-86fd7d7cbf24","Type":"ContainerStarted","Data":"fc7d267720968cb86005d8197aa7d37dfad9354a8d226a9f0f58d3f71c71f7c0"}
Sep 07 10:13:02.445149 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.445101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerStarted","Data":"42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9"}
Sep 07 10:13:02.445265 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.445244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerStarted","Data":"502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5"}
Sep 07 10:13:02.611482 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.611453    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:02.638627 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.638604    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:13:02.642167 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.642153    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.644928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.644852    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 07 10:13:02.644928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.644906    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 07 10:13:02.644928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.644907    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 07 10:13:02.645157 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.644957    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmqxd\""
Sep 07 10:13:02.645157 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.644858    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 07 10:13:02.645157 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.644906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 07 10:13:02.645157 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.644912    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 07 10:13:02.645575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.645548    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 07 10:13:02.645662 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.645590    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bsi6b934qt7tp\""
Sep 07 10:13:02.645662 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.645623    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 10:13:02.645662 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.645637    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 07 10:13:02.645980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.645681    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 07 10:13:02.645980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.645747    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 07 10:13:02.645980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.645833    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 07 10:13:02.646139 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.646023    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 07 10:13:02.646189 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.646140    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 07 10:13:02.647515 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.647496    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 07 10:13:02.657902 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.657884    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:13:02.686235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.686212    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:02.768827 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.768807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.768897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.768834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.768897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.768857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config-out\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.768897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.768884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.768913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.768958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.768993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlccc\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-kube-api-access-tlccc\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769462 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769462 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-web-config\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769462 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769462 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769462 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769462 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.769694 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.769472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870453 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870459    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870696 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870671    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870746 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870746 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870772    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config-out\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870801    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlccc\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-kube-api-access-tlccc\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870945    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.870982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.870971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.871255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.871039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.871255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.871093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.871255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.871128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.871255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.871163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.871255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.871189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-web-config\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.871508 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.871279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.871508 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.871377    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.873258 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.871684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.873670 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.873644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config-out\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.874256 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.873886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.874256 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.874229    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.874410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.874317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.874410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.874400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.875690 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.875059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.875690 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.875099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.875690 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.875272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-web-config\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.875690 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.875586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.875948 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.875709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.876113 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.876063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.876216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.876193    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.876283 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.876221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.876648 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.876629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.876832 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.876815    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.877026 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.877008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.886809 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.886786    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlccc\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-kube-api-access-tlccc\") pod \"prometheus-k8s-0\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:02.951078 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:02.951052    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:03.231621 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.231599    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:13:03.232860 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:13:03.232836    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511c2bb9_cc97_4f15_b6b9_2c25e5f621f4.slice/crio-6bc934df220f8bd5a56380944db29a6aa993f4c908496fd4d648cdce2c198c42 WatchSource:0}: Error finding container 6bc934df220f8bd5a56380944db29a6aa993f4c908496fd4d648cdce2c198c42: Status 404 returned error can't find the container with id 6bc934df220f8bd5a56380944db29a6aa993f4c908496fd4d648cdce2c198c42
Sep 07 10:13:03.452558 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.452527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6" event={"ID":"330f5d33-f755-4067-8b32-86fd7d7cbf24","Type":"ContainerStarted","Data":"bad1fc6ea821f29643f3d3ce3962b59c366bcd60904ae00977f90aef6e8d445c"}
Sep 07 10:13:03.452891 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.452607    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6" event={"ID":"330f5d33-f755-4067-8b32-86fd7d7cbf24","Type":"ContainerStarted","Data":"ac1796cb985280a4f49a5116404c5e8049d6a757629298f2be2467034ac3f76a"}
Sep 07 10:13:03.452891 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.452653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6" event={"ID":"330f5d33-f755-4067-8b32-86fd7d7cbf24","Type":"ContainerStarted","Data":"e0520bf43191826388b9541a2ae30fc6fc0591ced72c53a6406d7e543e03fec2"}
Sep 07 10:13:03.452891 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.452706    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:13:03.454245 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.454223    2560 generic.go:356] "Generic (PLEG): container finished" podID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605" exitCode=0
Sep 07 10:13:03.454350 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.454302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerDied","Data":"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"}
Sep 07 10:13:03.454392 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.454361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerStarted","Data":"6bc934df220f8bd5a56380944db29a6aa993f4c908496fd4d648cdce2c198c42"}
Sep 07 10:13:03.458171 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.458154    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerStarted","Data":"379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d"}
Sep 07 10:13:03.473692 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.473647    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6" podStartSLOduration=1.979529711 podStartE2EDuration="4.473633838s" totalImagesPullingTime="2.494104127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:03.471462905 +0000 UTC m=+90.019977888" watchObservedRunningTime="2026-09-07 10:13:03.473633838 +0000 UTC m=+90.022148825"
Sep 07 10:13:03.494914 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:03.494845    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.63941952 podStartE2EDuration="6.494832658s" totalImagesPullingTime="3.839437463s" totalInitContainerRuntime="15.975675ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:12:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:03.493549654 +0000 UTC m=+90.042064641" watchObservedRunningTime="2026-09-07 10:13:03.494832658 +0000 UTC m=+90.043347644"
Sep 07 10:13:05.041814 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:05.041756    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:05.141883 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:05.141845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:07.481627 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.481541    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerStarted","Data":"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"}
Sep 07 10:13:07.481627 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.481618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerStarted","Data":"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"}
Sep 07 10:13:07.482156 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.481658    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerStarted","Data":"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"}
Sep 07 10:13:07.482156 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.481702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerStarted","Data":"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"}
Sep 07 10:13:07.482156 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.481743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerStarted","Data":"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"}
Sep 07 10:13:07.482156 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.481790    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerStarted","Data":"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"}
Sep 07 10:13:07.493146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.493121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:07.507935 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.507863    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.094462452 podStartE2EDuration="5.507847939s" totalImagesPullingTime="3.395556781s" totalInitContainerRuntime="17.828706ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:07.506190397 +0000 UTC m=+94.054705385" watchObservedRunningTime="2026-09-07 10:13:07.507847939 +0000 UTC m=+94.056362929"
Sep 07 10:13:07.593468 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.593441    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:07.952044 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:07.952016    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:13:09.109258 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.109221    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:09.170174 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.170136    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:09.385990 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.385905    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76766cf945-v9bfw"]
Sep 07 10:13:09.390299 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.390277    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.400878 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.400857    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76766cf945-v9bfw"]
Sep 07 10:13:09.469084 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.469047    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-d856f4f5c-f98w6"
Sep 07 10:13:09.525872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.525844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7d5\" (UniqueName: \"kubernetes.io/projected/97149142-5e27-4cc5-89b2-f35caea58ab8-kube-api-access-wm7d5\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.526035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.525892    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-oauth-config\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.526035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.525925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-trusted-ca-bundle\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.526035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.525989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-oauth-serving-cert\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.526035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.526025    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-service-ca\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.526216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.526051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-serving-cert\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.526216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.526098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-console-config\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.626761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.626731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-oauth-serving-cert\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.626761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.626762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-service-ca\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.626949 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.626824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-serving-cert\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.627165 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.627139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-console-config\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.628141 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.627293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7d5\" (UniqueName: \"kubernetes.io/projected/97149142-5e27-4cc5-89b2-f35caea58ab8-kube-api-access-wm7d5\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.628141 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.627540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-oauth-config\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.628141 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.627544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-oauth-serving-cert\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.628141 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.627772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-console-config\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.628415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.628166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-trusted-ca-bundle\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.628599 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.628576    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-service-ca\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.629136 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.629115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-trusted-ca-bundle\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.629901 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.629870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-serving-cert\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.630179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.630160    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-oauth-config\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.634566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.634549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7d5\" (UniqueName: \"kubernetes.io/projected/97149142-5e27-4cc5-89b2-f35caea58ab8-kube-api-access-wm7d5\") pod \"console-76766cf945-v9bfw\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") " pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.699753 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.699702    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:09.972298 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:09.972273    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76766cf945-v9bfw"]
Sep 07 10:13:09.974799 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:13:09.974777    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97149142_5e27_4cc5_89b2_f35caea58ab8.slice/crio-0ec1c9e9f77377b8c8f65a62a5df2f8cc88bd590f76f4313970c4a4546fbd088 WatchSource:0}: Error finding container 0ec1c9e9f77377b8c8f65a62a5df2f8cc88bd590f76f4313970c4a4546fbd088: Status 404 returned error can't find the container with id 0ec1c9e9f77377b8c8f65a62a5df2f8cc88bd590f76f4313970c4a4546fbd088
Sep 07 10:13:10.496010 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:10.495974    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76766cf945-v9bfw" event={"ID":"97149142-5e27-4cc5-89b2-f35caea58ab8","Type":"ContainerStarted","Data":"23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928"}
Sep 07 10:13:10.496394 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:10.496037    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76766cf945-v9bfw" event={"ID":"97149142-5e27-4cc5-89b2-f35caea58ab8","Type":"ContainerStarted","Data":"0ec1c9e9f77377b8c8f65a62a5df2f8cc88bd590f76f4313970c4a4546fbd088"}
Sep 07 10:13:10.514228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:10.514184    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76766cf945-v9bfw" podStartSLOduration=1.514167868 podStartE2EDuration="1.514167868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:13:10.512924675 +0000 UTC m=+97.061439661" watchObservedRunningTime="2026-09-07 10:13:10.514167868 +0000 UTC m=+97.062682854"
Sep 07 10:13:19.367611 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:19.367582    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zzfpd"
Sep 07 10:13:19.700019 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:19.699939    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:19.700019 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:19.699999    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:19.704617 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:19.704596    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:20.040636 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:20.040605    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:20.140415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:20.140381    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:20.542427 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:20.542403    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:13:20.588063 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:20.588029    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b85794bc8-48l96"]
Sep 07 10:13:20.710737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:20.710705    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:20.811487 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:20.811400    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:20.866716 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:20.866693    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:25.493680 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.493639    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b4b557ff8-m6rm2" podUID="c853d585-23d4-4a0b-a96e-2dea59a9baec" containerName="console" containerID="cri-o://ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2" gracePeriod=15
Sep 07 10:13:25.735986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.735957    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:13:25.768045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.767422    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-service-ca\") pod \"c853d585-23d4-4a0b-a96e-2dea59a9baec\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") "
Sep 07 10:13:25.768045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.767476    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-oauth-serving-cert\") pod \"c853d585-23d4-4a0b-a96e-2dea59a9baec\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") "
Sep 07 10:13:25.768045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.767552    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-config\") pod \"c853d585-23d4-4a0b-a96e-2dea59a9baec\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") "
Sep 07 10:13:25.768045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.767585    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-serving-cert\") pod \"c853d585-23d4-4a0b-a96e-2dea59a9baec\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") "
Sep 07 10:13:25.768045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.767621    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-oauth-config\") pod \"c853d585-23d4-4a0b-a96e-2dea59a9baec\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") "
Sep 07 10:13:25.768045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.767682    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swwct\" (UniqueName: \"kubernetes.io/projected/c853d585-23d4-4a0b-a96e-2dea59a9baec-kube-api-access-swwct\") pod \"c853d585-23d4-4a0b-a96e-2dea59a9baec\" (UID: \"c853d585-23d4-4a0b-a96e-2dea59a9baec\") "
Sep 07 10:13:25.769162 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.768899    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-service-ca" (OuterVolumeSpecName: "service-ca") pod "c853d585-23d4-4a0b-a96e-2dea59a9baec" (UID: "c853d585-23d4-4a0b-a96e-2dea59a9baec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:13:25.769885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.769593    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-config" (OuterVolumeSpecName: "console-config") pod "c853d585-23d4-4a0b-a96e-2dea59a9baec" (UID: "c853d585-23d4-4a0b-a96e-2dea59a9baec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:13:25.770421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.770154    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c853d585-23d4-4a0b-a96e-2dea59a9baec" (UID: "c853d585-23d4-4a0b-a96e-2dea59a9baec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:13:25.785527 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.785221    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c853d585-23d4-4a0b-a96e-2dea59a9baec-kube-api-access-swwct" (OuterVolumeSpecName: "kube-api-access-swwct") pod "c853d585-23d4-4a0b-a96e-2dea59a9baec" (UID: "c853d585-23d4-4a0b-a96e-2dea59a9baec"). InnerVolumeSpecName "kube-api-access-swwct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:13:25.785527 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.785239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c853d585-23d4-4a0b-a96e-2dea59a9baec" (UID: "c853d585-23d4-4a0b-a96e-2dea59a9baec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:13:25.785527 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.785303    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c853d585-23d4-4a0b-a96e-2dea59a9baec" (UID: "c853d585-23d4-4a0b-a96e-2dea59a9baec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:13:25.868656 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.868614    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swwct\" (UniqueName: \"kubernetes.io/projected/c853d585-23d4-4a0b-a96e-2dea59a9baec-kube-api-access-swwct\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:25.868656 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.868652    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:25.868656 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.868662    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-oauth-serving-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:25.868656 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.868673    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:25.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.868682    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-serving-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:25.868879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:25.868690    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c853d585-23d4-4a0b-a96e-2dea59a9baec-console-oauth-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:26.535182 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.535151    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:26.562529 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.562500    2560 generic.go:356] "Generic (PLEG): container finished" podID="c853d585-23d4-4a0b-a96e-2dea59a9baec" containerID="ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2" exitCode=0
Sep 07 10:13:26.562654 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.562547    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4b557ff8-m6rm2"
Sep 07 10:13:26.562654 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.562565    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b557ff8-m6rm2" event={"ID":"c853d585-23d4-4a0b-a96e-2dea59a9baec","Type":"ContainerDied","Data":"ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2"}
Sep 07 10:13:26.562654 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.562613    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4b557ff8-m6rm2" event={"ID":"c853d585-23d4-4a0b-a96e-2dea59a9baec","Type":"ContainerDied","Data":"a0cb93056027175a873f7d682f15b559a8e805bbca3c65e0c78818e56eda557f"}
Sep 07 10:13:26.562654 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.562633    2560 scope.go:117] "RemoveContainer" containerID="ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2"
Sep 07 10:13:26.570722 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.570700    2560 scope.go:117] "RemoveContainer" containerID="ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2"
Sep 07 10:13:26.570972 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:13:26.570951    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2\": container with ID starting with ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2 not found: ID does not exist" containerID="ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2"
Sep 07 10:13:26.571021 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.570982    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2"} err="failed to get container status \"ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2\": rpc error: code = NotFound desc = could not find container \"ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2\": container with ID starting with ba0b8a31b2e59f5edc06c297a831efad4fd00571cd6cef715690c7fa56bbf1b2 not found: ID does not exist"
Sep 07 10:13:26.579724 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.579700    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4b557ff8-m6rm2"]
Sep 07 10:13:26.585440 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.585421    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b4b557ff8-m6rm2"]
Sep 07 10:13:26.598597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.598565    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:26.659312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:26.659288    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:27.894013 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:27.893981    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c853d585-23d4-4a0b-a96e-2dea59a9baec" path="/var/lib/kubelet/pods/c853d585-23d4-4a0b-a96e-2dea59a9baec/volumes"
Sep 07 10:13:31.413494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:31.413463    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:31.515846 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:31.515814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:13:45.607775 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.607724    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b85794bc8-48l96" podUID="ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" containerName="console" containerID="cri-o://113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1" gracePeriod=15
Sep 07 10:13:45.842698 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.842676    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:13:45.928235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928176    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-serving-cert\") pod \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") "
Sep 07 10:13:45.928235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928206    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-config\") pod \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") "
Sep 07 10:13:45.928235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928227    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-oauth-config\") pod \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") "
Sep 07 10:13:45.928464 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-trusted-ca-bundle\") pod \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") "
Sep 07 10:13:45.928464 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldgxx\" (UniqueName: \"kubernetes.io/projected/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-kube-api-access-ldgxx\") pod \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") "
Sep 07 10:13:45.928464 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928327    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-service-ca\") pod \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") "
Sep 07 10:13:45.928464 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928386    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-oauth-serving-cert\") pod \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\" (UID: \"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3\") "
Sep 07 10:13:45.928694 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928665    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-config" (OuterVolumeSpecName: "console-config") pod "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" (UID: "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:13:45.928753 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928688    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" (UID: "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:13:45.928753 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928733    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-service-ca" (OuterVolumeSpecName: "service-ca") pod "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" (UID: "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:13:45.928957 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.928934    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" (UID: "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:13:45.930328 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.930308    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" (UID: "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:13:45.930454 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.930430    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" (UID: "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:13:45.930551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:45.930484    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-kube-api-access-ldgxx" (OuterVolumeSpecName: "kube-api-access-ldgxx") pod "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" (UID: "ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3"). InnerVolumeSpecName "kube-api-access-ldgxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:13:46.029138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.029105    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-serving-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:46.029138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.029133    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:46.029278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.029151    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-console-oauth-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:46.029278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.029164    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-trusted-ca-bundle\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:46.029278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.029178    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldgxx\" (UniqueName: \"kubernetes.io/projected/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-kube-api-access-ldgxx\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:46.029278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.029189    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:46.029278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.029197    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3-oauth-serving-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:13:46.648110 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.648065    2560 generic.go:356] "Generic (PLEG): container finished" podID="ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" containerID="113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1" exitCode=0
Sep 07 10:13:46.648491 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.648125    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b85794bc8-48l96"
Sep 07 10:13:46.648491 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.648154    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b85794bc8-48l96" event={"ID":"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3","Type":"ContainerDied","Data":"113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1"}
Sep 07 10:13:46.648491 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.648193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b85794bc8-48l96" event={"ID":"ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3","Type":"ContainerDied","Data":"6a072a2dd25b630e7b0d357c5bf7c52649e3390357701f6d244d55e6a2f396e1"}
Sep 07 10:13:46.648491 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.648211    2560 scope.go:117] "RemoveContainer" containerID="113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1"
Sep 07 10:13:46.656528 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.656512    2560 scope.go:117] "RemoveContainer" containerID="113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1"
Sep 07 10:13:46.656797 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:13:46.656779    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1\": container with ID starting with 113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1 not found: ID does not exist" containerID="113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1"
Sep 07 10:13:46.656845 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.656805    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1"} err="failed to get container status \"113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1\": rpc error: code = NotFound desc = could not find container \"113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1\": container with ID starting with 113be094f0458a52556783d856a0517c1fc745d7654facae925fac315ea254b1 not found: ID does not exist"
Sep 07 10:13:46.667425 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.667399    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b85794bc8-48l96"]
Sep 07 10:13:46.670747 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:46.670723    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b85794bc8-48l96"]
Sep 07 10:13:47.894697 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:13:47.894660    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" path="/var/lib/kubelet/pods/ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3/volumes"
Sep 07 10:14:02.952139 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:02.952104    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:02.970575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:02.970554    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:03.731303 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:03.731274    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:12.512774 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:12.512742    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:12.565712 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:12.565682    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:16.740038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:16.740003    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 10:14:16.740532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:16.740335    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="alertmanager" containerID="cri-o://f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd" gracePeriod=120
Sep 07 10:14:16.740532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:16.740379    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy-metric" containerID="cri-o://42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9" gracePeriod=120
Sep 07 10:14:16.740532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:16.740423    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy-web" containerID="cri-o://c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753" gracePeriod=120
Sep 07 10:14:16.740532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:16.740454    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy" containerID="cri-o://502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5" gracePeriod=120
Sep 07 10:14:16.740532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:16.740448    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="config-reloader" containerID="cri-o://a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091" gracePeriod=120
Sep 07 10:14:16.740532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:16.740516    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="prom-label-proxy" containerID="cri-o://379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d" gracePeriod=120
Sep 07 10:14:17.774290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774252    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerID="379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d" exitCode=0
Sep 07 10:14:17.774290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774280    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerID="42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9" exitCode=0
Sep 07 10:14:17.774290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774289    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerID="502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5" exitCode=0
Sep 07 10:14:17.774290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774297    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerID="a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091" exitCode=0
Sep 07 10:14:17.774744 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774305    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerID="f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd" exitCode=0
Sep 07 10:14:17.774744 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerDied","Data":"379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d"}
Sep 07 10:14:17.774744 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerDied","Data":"42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9"}
Sep 07 10:14:17.774744 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerDied","Data":"502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5"}
Sep 07 10:14:17.774744 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerDied","Data":"a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091"}
Sep 07 10:14:17.774744 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.774468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerDied","Data":"f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd"}
Sep 07 10:14:17.978261 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:17.978239    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:14:18.067263 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067404 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067284    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-metrics-client-ca\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067404 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067310    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-web-config\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067404 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067328    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-tls-assets\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067404 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067345    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-main-tls\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067404 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067365    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-trusted-ca-bundle\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067431    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-out\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5w66\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-kube-api-access-t5w66\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-main-db\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067527    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-volume\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067550    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067576    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-cluster-tls-config\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.067678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067616    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\" (UID: \"2c9cb02c-8b85-464d-b012-0a8ae11346f3\") "
Sep 07 10:14:18.068010 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067710    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:18.068010 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.067911    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-metrics-client-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.068236 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.068186    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:18.068809 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.068766    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:14:18.070561 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.070534    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:14:18.070663 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.070633    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:18.071021 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.070992    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:18.071021 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.071008    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-volume" (OuterVolumeSpecName: "config-volume") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:18.071190 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.071136    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:18.071244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.071205    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-kube-api-access-t5w66" (OuterVolumeSpecName: "kube-api-access-t5w66") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "kube-api-access-t5w66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:14:18.071244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.071210    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-out" (OuterVolumeSpecName: "config-out") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:14:18.071851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.071831    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:18.074831 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.074812    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:18.081286 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.081266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-web-config" (OuterVolumeSpecName: "web-config") pod "2c9cb02c-8b85-464d-b012-0a8ae11346f3" (UID: "2c9cb02c-8b85-464d-b012-0a8ae11346f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:18.168914 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.168881    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.168933    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-web-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.168951    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-tls-assets\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.168961    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-main-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.168970    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.168978    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-out\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.168987    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5w66\" (UniqueName: \"kubernetes.io/projected/2c9cb02c-8b85-464d-b012-0a8ae11346f3-kube-api-access-t5w66\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.168996    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c9cb02c-8b85-464d-b012-0a8ae11346f3-alertmanager-main-db\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.169004    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.169012    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.169020    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-cluster-tls-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.169260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.169029    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c9cb02c-8b85-464d-b012-0a8ae11346f3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:18.780822 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.780785    2560 generic.go:356] "Generic (PLEG): container finished" podID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerID="c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753" exitCode=0
Sep 07 10:14:18.781200 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.780847    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerDied","Data":"c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753"}
Sep 07 10:14:18.781200 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.780877    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 07 10:14:18.781200 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.780928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c9cb02c-8b85-464d-b012-0a8ae11346f3","Type":"ContainerDied","Data":"28f887d0b09ae52c2ac9b7e301b4dc7526a01a4c73fcf1865c044a333c57fd50"}
Sep 07 10:14:18.781200 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.780962    2560 scope.go:117] "RemoveContainer" containerID="379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d"
Sep 07 10:14:18.788871 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.788854    2560 scope.go:117] "RemoveContainer" containerID="42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9"
Sep 07 10:14:18.795588 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.795571    2560 scope.go:117] "RemoveContainer" containerID="502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5"
Sep 07 10:14:18.802666 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.802619    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 10:14:18.805159 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.804632    2560 scope.go:117] "RemoveContainer" containerID="c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753"
Sep 07 10:14:18.805733 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.805711    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 07 10:14:18.811711 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.811695    2560 scope.go:117] "RemoveContainer" containerID="a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091"
Sep 07 10:14:18.818213 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.818191    2560 scope.go:117] "RemoveContainer" containerID="f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd"
Sep 07 10:14:18.824506 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.824491    2560 scope.go:117] "RemoveContainer" containerID="2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8"
Sep 07 10:14:18.830776 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.830764    2560 scope.go:117] "RemoveContainer" containerID="379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d"
Sep 07 10:14:18.831005 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:18.830988    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d\": container with ID starting with 379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d not found: ID does not exist" containerID="379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d"
Sep 07 10:14:18.831057 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.831011    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d"} err="failed to get container status \"379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d\": rpc error: code = NotFound desc = could not find container \"379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d\": container with ID starting with 379d92c713a67b85668a08938b82f2927dedff39dd26713e5ab7f74b13619f4d not found: ID does not exist"
Sep 07 10:14:18.831057 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.831026    2560 scope.go:117] "RemoveContainer" containerID="42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9"
Sep 07 10:14:18.831267 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:18.831248    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9\": container with ID starting with 42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9 not found: ID does not exist" containerID="42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9"
Sep 07 10:14:18.831304 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.831275    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9"} err="failed to get container status \"42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9\": rpc error: code = NotFound desc = could not find container \"42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9\": container with ID starting with 42bc5242169b0873ba49d2a76261d8c4360918a61db06f47e373537fbea83bf9 not found: ID does not exist"
Sep 07 10:14:18.831304 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.831292    2560 scope.go:117] "RemoveContainer" containerID="502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5"
Sep 07 10:14:18.831544 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:18.831530    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5\": container with ID starting with 502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5 not found: ID does not exist" containerID="502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5"
Sep 07 10:14:18.831580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.831548    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5"} err="failed to get container status \"502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5\": rpc error: code = NotFound desc = could not find container \"502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5\": container with ID starting with 502f421023cb251d3ec1b5f1023a70396300ea70c041b20bb803ab3979ef27f5 not found: ID does not exist"
Sep 07 10:14:18.831580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.831567    2560 scope.go:117] "RemoveContainer" containerID="c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753"
Sep 07 10:14:18.831757 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:18.831742    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753\": container with ID starting with c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753 not found: ID does not exist" containerID="c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753"
Sep 07 10:14:18.831794 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.831760    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753"} err="failed to get container status \"c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753\": rpc error: code = NotFound desc = could not find container \"c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753\": container with ID starting with c7457cdfc526530b1e72ed4a3151f953b62c1575371fa41b57febef32b7ad753 not found: ID does not exist"
Sep 07 10:14:18.831794 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.831773    2560 scope.go:117] "RemoveContainer" containerID="a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091"
Sep 07 10:14:18.832013 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:18.831996    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091\": container with ID starting with a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091 not found: ID does not exist" containerID="a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091"
Sep 07 10:14:18.832056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.832019    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091"} err="failed to get container status \"a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091\": rpc error: code = NotFound desc = could not find container \"a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091\": container with ID starting with a93d37e8a470176ce2426a54333f7351546c63d0af4673a94826c5a630b96091 not found: ID does not exist"
Sep 07 10:14:18.832056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.832032    2560 scope.go:117] "RemoveContainer" containerID="f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd"
Sep 07 10:14:18.832286 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:18.832268    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd\": container with ID starting with f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd not found: ID does not exist" containerID="f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd"
Sep 07 10:14:18.832340 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.832288    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd"} err="failed to get container status \"f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd\": rpc error: code = NotFound desc = could not find container \"f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd\": container with ID starting with f0e5b656d2898561fc30953bcfe120f017ba82d83160f3178eae0b974d24d4cd not found: ID does not exist"
Sep 07 10:14:18.832340 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.832302    2560 scope.go:117] "RemoveContainer" containerID="2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8"
Sep 07 10:14:18.832558 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:18.832542    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8\": container with ID starting with 2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8 not found: ID does not exist" containerID="2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8"
Sep 07 10:14:18.832596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:18.832565    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8"} err="failed to get container status \"2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8\": rpc error: code = NotFound desc = could not find container \"2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8\": container with ID starting with 2cdc1b48181bf174871c63af43394cc4405a113ba9b9b9355a7b4bb0e673a4f8 not found: ID does not exist"
Sep 07 10:14:19.899024 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:19.898991    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" path="/var/lib/kubelet/pods/2c9cb02c-8b85-464d-b012-0a8ae11346f3/volumes"
Sep 07 10:14:20.967095 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:20.967043    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:14:20.967497 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:20.967360    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="config-reloader" containerID="cri-o://7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e" gracePeriod=600
Sep 07 10:14:20.967497 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:20.967385    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="prometheus" containerID="cri-o://84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002" gracePeriod=600
Sep 07 10:14:20.967497 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:20.967425    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy-web" containerID="cri-o://bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8" gracePeriod=600
Sep 07 10:14:20.967497 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:20.967381    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="thanos-sidecar" containerID="cri-o://32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6" gracePeriod=600
Sep 07 10:14:20.967497 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:20.967358    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy" containerID="cri-o://d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2" gracePeriod=600
Sep 07 10:14:20.967733 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:20.967428    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983" gracePeriod=600
Sep 07 10:14:21.201451 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.201427    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:21.395341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395299    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-web-config\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395364    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-kubelet-serving-ca-bundle\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395393    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-tls-assets\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395415    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-grpc-tls\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395455    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395481    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlccc\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-kube-api-access-tlccc\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395504    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-metrics-client-ca\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395778 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395737    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config-out\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395834 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395774    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395834 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395809    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-metrics-client-certs\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395951 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395835    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-trusted-ca-bundle\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395951 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395854    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:21.395951 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395873    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-serving-certs-ca-bundle\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395951 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395904    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-0\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.395951 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395929    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.396216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395972    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-tls\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.396216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395997    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-kube-rbac-proxy\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.396216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.396022    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-db\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.396216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.396096    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-1\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.396216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.396122    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-thanos-prometheus-http-client-file\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.396216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.396178    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-2\") pod \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\" (UID: \"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4\") "
Sep 07 10:14:21.396561 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.396453    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.395968    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.398660    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.396372    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.396900    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.397927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.398034    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.398091    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-kube-api-access-tlccc" (OuterVolumeSpecName: "kube-api-access-tlccc") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "kube-api-access-tlccc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.398259    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config" (OuterVolumeSpecName: "config") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.398362    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:21.398885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.398639    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:21.399504 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.398912    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.399897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.399869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:14:21.400004 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.399962    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:14:21.400123 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.400096    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.400209 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.400186    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config-out" (OuterVolumeSpecName: "config-out") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:14:21.400432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.400398    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.401421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.401394    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.401545 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.401527    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.407715 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.407696    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-web-config" (OuterVolumeSpecName: "web-config") pod "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" (UID: "511c2bb9-cc97-4f15-b6b9-2c25e5f621f4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:14:21.497745 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497717    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497745 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497745    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlccc\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-kube-api-access-tlccc\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497756    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-metrics-client-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497766    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config-out\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497774    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497781    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-metrics-client-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497789    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497797    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497806    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497814    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497823    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497831    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-kube-rbac-proxy\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497838    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-db\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497846    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.497856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497854    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.498269 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497862    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.498269 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497890    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-web-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.498269 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497898    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-tls-assets\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.498269 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.497906    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4-secret-grpc-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:14:21.796292 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796264    2560 generic.go:356] "Generic (PLEG): container finished" podID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983" exitCode=0
Sep 07 10:14:21.796292 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796286    2560 generic.go:356] "Generic (PLEG): container finished" podID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2" exitCode=0
Sep 07 10:14:21.796292 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796292    2560 generic.go:356] "Generic (PLEG): container finished" podID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8" exitCode=0
Sep 07 10:14:21.796292 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796298    2560 generic.go:356] "Generic (PLEG): container finished" podID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6" exitCode=0
Sep 07 10:14:21.796524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796303    2560 generic.go:356] "Generic (PLEG): container finished" podID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e" exitCode=0
Sep 07 10:14:21.796524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796320    2560 generic.go:356] "Generic (PLEG): container finished" podID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002" exitCode=0
Sep 07 10:14:21.796524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796355    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerDied","Data":"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"}
Sep 07 10:14:21.796524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796425    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerDied","Data":"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"}
Sep 07 10:14:21.796524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796367    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:21.796524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796466    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerDied","Data":"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"}
Sep 07 10:14:21.796524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796506    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerDied","Data":"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"}
Sep 07 10:14:21.796768 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796535    2560 scope.go:117] "RemoveContainer" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"
Sep 07 10:14:21.796768 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerDied","Data":"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"}
Sep 07 10:14:21.796768 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796587    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerDied","Data":"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"}
Sep 07 10:14:21.796768 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.796630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"511c2bb9-cc97-4f15-b6b9-2c25e5f621f4","Type":"ContainerDied","Data":"6bc934df220f8bd5a56380944db29a6aa993f4c908496fd4d648cdce2c198c42"}
Sep 07 10:14:21.804499 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.804482    2560 scope.go:117] "RemoveContainer" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"
Sep 07 10:14:21.811770 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.811753    2560 scope.go:117] "RemoveContainer" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"
Sep 07 10:14:21.818367 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.818345    2560 scope.go:117] "RemoveContainer" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"
Sep 07 10:14:21.820141 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.820116    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:14:21.824769 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.824741    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:14:21.826289 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.826274    2560 scope.go:117] "RemoveContainer" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"
Sep 07 10:14:21.832891 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.832875    2560 scope.go:117] "RemoveContainer" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"
Sep 07 10:14:21.839599 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.839584    2560 scope.go:117] "RemoveContainer" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"
Sep 07 10:14:21.845972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.845956    2560 scope.go:117] "RemoveContainer" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"
Sep 07 10:14:21.846268 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:21.846248    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": container with ID starting with a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983 not found: ID does not exist" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"
Sep 07 10:14:21.846342 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.846278    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"} err="failed to get container status \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": rpc error: code = NotFound desc = could not find container \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": container with ID starting with a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983 not found: ID does not exist"
Sep 07 10:14:21.846342 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.846304    2560 scope.go:117] "RemoveContainer" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"
Sep 07 10:14:21.846569 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:21.846550    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": container with ID starting with d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2 not found: ID does not exist" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"
Sep 07 10:14:21.846607 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.846575    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"} err="failed to get container status \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": rpc error: code = NotFound desc = could not find container \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": container with ID starting with d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2 not found: ID does not exist"
Sep 07 10:14:21.846607 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.846594    2560 scope.go:117] "RemoveContainer" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"
Sep 07 10:14:21.846825 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:21.846808    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": container with ID starting with bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8 not found: ID does not exist" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"
Sep 07 10:14:21.846889 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.846833    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"} err="failed to get container status \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": rpc error: code = NotFound desc = could not find container \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": container with ID starting with bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8 not found: ID does not exist"
Sep 07 10:14:21.846889 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.846854    2560 scope.go:117] "RemoveContainer" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"
Sep 07 10:14:21.847088 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:21.847056    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": container with ID starting with 32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6 not found: ID does not exist" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"
Sep 07 10:14:21.847146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.847094    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"} err="failed to get container status \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": rpc error: code = NotFound desc = could not find container \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": container with ID starting with 32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6 not found: ID does not exist"
Sep 07 10:14:21.847146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.847108    2560 scope.go:117] "RemoveContainer" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"
Sep 07 10:14:21.847336 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:21.847322    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": container with ID starting with 7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e not found: ID does not exist" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"
Sep 07 10:14:21.847381 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.847338    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"} err="failed to get container status \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": rpc error: code = NotFound desc = could not find container \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": container with ID starting with 7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e not found: ID does not exist"
Sep 07 10:14:21.847381 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.847349    2560 scope.go:117] "RemoveContainer" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"
Sep 07 10:14:21.847763 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:21.847733    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": container with ID starting with 84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002 not found: ID does not exist" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"
Sep 07 10:14:21.848444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.847764    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"} err="failed to get container status \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": rpc error: code = NotFound desc = could not find container \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": container with ID starting with 84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002 not found: ID does not exist"
Sep 07 10:14:21.848444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.847784    2560 scope.go:117] "RemoveContainer" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"
Sep 07 10:14:21.848444 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:14:21.848048    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": container with ID starting with 38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605 not found: ID does not exist" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"
Sep 07 10:14:21.848444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.848092    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"} err="failed to get container status \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": rpc error: code = NotFound desc = could not find container \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": container with ID starting with 38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605 not found: ID does not exist"
Sep 07 10:14:21.848444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.848113    2560 scope.go:117] "RemoveContainer" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"
Sep 07 10:14:21.848444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.848370    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"} err="failed to get container status \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": rpc error: code = NotFound desc = could not find container \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": container with ID starting with a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983 not found: ID does not exist"
Sep 07 10:14:21.848444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.848390    2560 scope.go:117] "RemoveContainer" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"
Sep 07 10:14:21.848821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.848652    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"} err="failed to get container status \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": rpc error: code = NotFound desc = could not find container \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": container with ID starting with d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2 not found: ID does not exist"
Sep 07 10:14:21.848821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.848676    2560 scope.go:117] "RemoveContainer" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"
Sep 07 10:14:21.849002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.848974    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"} err="failed to get container status \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": rpc error: code = NotFound desc = could not find container \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": container with ID starting with bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8 not found: ID does not exist"
Sep 07 10:14:21.849002 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849002    2560 scope.go:117] "RemoveContainer" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"
Sep 07 10:14:21.849306 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849283    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"} err="failed to get container status \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": rpc error: code = NotFound desc = could not find container \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": container with ID starting with 32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6 not found: ID does not exist"
Sep 07 10:14:21.849306 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849307    2560 scope.go:117] "RemoveContainer" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"
Sep 07 10:14:21.849588 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849565    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"} err="failed to get container status \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": rpc error: code = NotFound desc = could not find container \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": container with ID starting with 7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e not found: ID does not exist"
Sep 07 10:14:21.849588 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849587    2560 scope.go:117] "RemoveContainer" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"
Sep 07 10:14:21.849685 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849586    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:14:21.849861 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849841    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"} err="failed to get container status \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": rpc error: code = NotFound desc = could not find container \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": container with ID starting with 84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002 not found: ID does not exist"
Sep 07 10:14:21.849929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849863    2560 scope.go:117] "RemoveContainer" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"
Sep 07 10:14:21.849929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849885    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="thanos-sidecar"
Sep 07 10:14:21.849929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849897    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="thanos-sidecar"
Sep 07 10:14:21.849929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849906    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="alertmanager"
Sep 07 10:14:21.849929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849912    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="alertmanager"
Sep 07 10:14:21.849929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849918    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="config-reloader"
Sep 07 10:14:21.849929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849925    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="config-reloader"
Sep 07 10:14:21.849929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849931    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy-thanos"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849937    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy-thanos"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849949    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="prom-label-proxy"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849957    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="prom-label-proxy"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849962    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="init-config-reloader"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849968    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="init-config-reloader"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849973    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849978    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849984    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="config-reloader"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849989    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="config-reloader"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.849995    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="init-config-reloader"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850001    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="init-config-reloader"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850007    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="prometheus"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850012    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="prometheus"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850018    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c853d585-23d4-4a0b-a96e-2dea59a9baec" containerName="console"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850023    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c853d585-23d4-4a0b-a96e-2dea59a9baec" containerName="console"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850030    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy-metric"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850035    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy-metric"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850044    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy-web"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850052    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy-web"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850062    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy-web"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850091    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy-web"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850104    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" containerName="console"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850112    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" containerName="console"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850121    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850118    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"} err="failed to get container status \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": rpc error: code = NotFound desc = could not find container \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": container with ID starting with 38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605 not found: ID does not exist"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850139    2560 scope.go:117] "RemoveContainer" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"
Sep 07 10:14:21.850226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850129    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850253    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec24e20b-60aa-43c0-8c11-0ff9b1e5fef3" containerName="console"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850266    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy-web"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850274    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy-web"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850286    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="alertmanager"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850293    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy-thanos"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850302    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="config-reloader"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850312    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="config-reloader"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850317    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="prometheus"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850322    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="thanos-sidecar"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850330    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c853d585-23d4-4a0b-a96e-2dea59a9baec" containerName="console"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850338    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy-metric"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850348    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="prom-label-proxy"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850356    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" containerName="kube-rbac-proxy"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850363    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9cb02c-8b85-464d-b012-0a8ae11346f3" containerName="kube-rbac-proxy"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850362    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"} err="failed to get container status \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": rpc error: code = NotFound desc = could not find container \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": container with ID starting with a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983 not found: ID does not exist"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850380    2560 scope.go:117] "RemoveContainer" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850606    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"} err="failed to get container status \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": rpc error: code = NotFound desc = could not find container \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": container with ID starting with d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2 not found: ID does not exist"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850631    2560 scope.go:117] "RemoveContainer" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850907    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"} err="failed to get container status \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": rpc error: code = NotFound desc = could not find container \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": container with ID starting with bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8 not found: ID does not exist"
Sep 07 10:14:21.851146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.850926    2560 scope.go:117] "RemoveContainer" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"
Sep 07 10:14:21.851928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.851189    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"} err="failed to get container status \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": rpc error: code = NotFound desc = could not find container \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": container with ID starting with 32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6 not found: ID does not exist"
Sep 07 10:14:21.851928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.851202    2560 scope.go:117] "RemoveContainer" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"
Sep 07 10:14:21.851928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.851440    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"} err="failed to get container status \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": rpc error: code = NotFound desc = could not find container \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": container with ID starting with 7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e not found: ID does not exist"
Sep 07 10:14:21.851928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.851461    2560 scope.go:117] "RemoveContainer" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"
Sep 07 10:14:21.851928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.851668    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"} err="failed to get container status \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": rpc error: code = NotFound desc = could not find container \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": container with ID starting with 84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002 not found: ID does not exist"
Sep 07 10:14:21.851928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.851686    2560 scope.go:117] "RemoveContainer" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"
Sep 07 10:14:21.851928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.851898    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"} err="failed to get container status \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": rpc error: code = NotFound desc = could not find container \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": container with ID starting with 38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605 not found: ID does not exist"
Sep 07 10:14:21.851928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.851915    2560 scope.go:117] "RemoveContainer" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"
Sep 07 10:14:21.852215 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.852133    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"} err="failed to get container status \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": rpc error: code = NotFound desc = could not find container \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": container with ID starting with a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983 not found: ID does not exist"
Sep 07 10:14:21.852215 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.852151    2560 scope.go:117] "RemoveContainer" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"
Sep 07 10:14:21.852398 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.852380    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"} err="failed to get container status \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": rpc error: code = NotFound desc = could not find container \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": container with ID starting with d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2 not found: ID does not exist"
Sep 07 10:14:21.852448 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.852399    2560 scope.go:117] "RemoveContainer" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"
Sep 07 10:14:21.852660 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.852639    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"} err="failed to get container status \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": rpc error: code = NotFound desc = could not find container \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": container with ID starting with bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8 not found: ID does not exist"
Sep 07 10:14:21.852714 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.852660    2560 scope.go:117] "RemoveContainer" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"
Sep 07 10:14:21.852915 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.852896    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"} err="failed to get container status \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": rpc error: code = NotFound desc = could not find container \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": container with ID starting with 32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6 not found: ID does not exist"
Sep 07 10:14:21.852957 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.852924    2560 scope.go:117] "RemoveContainer" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"
Sep 07 10:14:21.853173 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.853155    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"} err="failed to get container status \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": rpc error: code = NotFound desc = could not find container \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": container with ID starting with 7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e not found: ID does not exist"
Sep 07 10:14:21.853223 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.853176    2560 scope.go:117] "RemoveContainer" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"
Sep 07 10:14:21.853420 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.853403    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"} err="failed to get container status \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": rpc error: code = NotFound desc = could not find container \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": container with ID starting with 84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002 not found: ID does not exist"
Sep 07 10:14:21.853463 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.853421    2560 scope.go:117] "RemoveContainer" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"
Sep 07 10:14:21.853613 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.853599    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"} err="failed to get container status \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": rpc error: code = NotFound desc = could not find container \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": container with ID starting with 38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605 not found: ID does not exist"
Sep 07 10:14:21.853661 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.853612    2560 scope.go:117] "RemoveContainer" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"
Sep 07 10:14:21.853821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.853806    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"} err="failed to get container status \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": rpc error: code = NotFound desc = could not find container \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": container with ID starting with a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983 not found: ID does not exist"
Sep 07 10:14:21.853872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.853823    2560 scope.go:117] "RemoveContainer" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"
Sep 07 10:14:21.854017 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854002    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"} err="failed to get container status \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": rpc error: code = NotFound desc = could not find container \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": container with ID starting with d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2 not found: ID does not exist"
Sep 07 10:14:21.854052 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854016    2560 scope.go:117] "RemoveContainer" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"
Sep 07 10:14:21.854234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854219    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"} err="failed to get container status \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": rpc error: code = NotFound desc = could not find container \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": container with ID starting with bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8 not found: ID does not exist"
Sep 07 10:14:21.854276 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854236    2560 scope.go:117] "RemoveContainer" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"
Sep 07 10:14:21.854430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854416    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"} err="failed to get container status \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": rpc error: code = NotFound desc = could not find container \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": container with ID starting with 32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6 not found: ID does not exist"
Sep 07 10:14:21.854465 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854429    2560 scope.go:117] "RemoveContainer" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"
Sep 07 10:14:21.854628 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854615    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"} err="failed to get container status \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": rpc error: code = NotFound desc = could not find container \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": container with ID starting with 7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e not found: ID does not exist"
Sep 07 10:14:21.854661 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854629    2560 scope.go:117] "RemoveContainer" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"
Sep 07 10:14:21.854816 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854802    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"} err="failed to get container status \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": rpc error: code = NotFound desc = could not find container \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": container with ID starting with 84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002 not found: ID does not exist"
Sep 07 10:14:21.854852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.854816    2560 scope.go:117] "RemoveContainer" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"
Sep 07 10:14:21.855033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.855016    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"} err="failed to get container status \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": rpc error: code = NotFound desc = could not find container \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": container with ID starting with 38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605 not found: ID does not exist"
Sep 07 10:14:21.855088 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.855033    2560 scope.go:117] "RemoveContainer" containerID="a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"
Sep 07 10:14:21.855277 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.855262    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983"} err="failed to get container status \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": rpc error: code = NotFound desc = could not find container \"a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983\": container with ID starting with a4d24a13cdc945f72abe8c9c662270d24950f569374cb9b4d30ee457ce064983 not found: ID does not exist"
Sep 07 10:14:21.855277 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.855272    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:21.855366 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.855276    2560 scope.go:117] "RemoveContainer" containerID="d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"
Sep 07 10:14:21.855626 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.855603    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2"} err="failed to get container status \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": rpc error: code = NotFound desc = could not find container \"d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2\": container with ID starting with d82023b364cdbbfebd3d6398cf19544248ede2583645d8957b8595538be8bad2 not found: ID does not exist"
Sep 07 10:14:21.855707 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.855628    2560 scope.go:117] "RemoveContainer" containerID="bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"
Sep 07 10:14:21.856020 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.855993    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8"} err="failed to get container status \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": rpc error: code = NotFound desc = could not find container \"bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8\": container with ID starting with bca4cd45f1b2de9599cf6aa01d72ae7e4860015dce54b0e01d74fb73997db7e8 not found: ID does not exist"
Sep 07 10:14:21.856103 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.856024    2560 scope.go:117] "RemoveContainer" containerID="32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"
Sep 07 10:14:21.856302 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.856283    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6"} err="failed to get container status \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": rpc error: code = NotFound desc = could not find container \"32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6\": container with ID starting with 32edfe947d7a681a78ac05971ae71aaed461ba39c1ccb6a186376705dcf40ce6 not found: ID does not exist"
Sep 07 10:14:21.856365 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.856301    2560 scope.go:117] "RemoveContainer" containerID="7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"
Sep 07 10:14:21.856561 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.856541    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e"} err="failed to get container status \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": rpc error: code = NotFound desc = could not find container \"7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e\": container with ID starting with 7868d9cc6ba829bdbd1892c46f675e4a522b4bbae44ddc4c4524b78532f9311e not found: ID does not exist"
Sep 07 10:14:21.856561 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.856559    2560 scope.go:117] "RemoveContainer" containerID="84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"
Sep 07 10:14:21.856811 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.856794    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002"} err="failed to get container status \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": rpc error: code = NotFound desc = could not find container \"84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002\": container with ID starting with 84676d12954f4d8dec53c0c95eb647b412ee9cddcc230f868c5e05511407b002 not found: ID does not exist"
Sep 07 10:14:21.856883 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.856812    2560 scope.go:117] "RemoveContainer" containerID="38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"
Sep 07 10:14:21.857045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.857027    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605"} err="failed to get container status \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": rpc error: code = NotFound desc = could not find container \"38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605\": container with ID starting with 38af17cdef93d925c1f72e1c76232e145d39e842aa364c23a3a08485087c1605 not found: ID does not exist"
Sep 07 10:14:21.857766 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.857747    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 07 10:14:21.857908 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.857863    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 07 10:14:21.857908 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.857871    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 07 10:14:21.858149 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858135    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 07 10:14:21.858250 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858233    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 10:14:21.858332 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858249    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 07 10:14:21.858332 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858296    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 07 10:14:21.858332 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858232    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 07 10:14:21.858332 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858326    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 07 10:14:21.858332 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858300    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 07 10:14:21.858851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858458    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 07 10:14:21.858851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858577    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bsi6b934qt7tp\""
Sep 07 10:14:21.858851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858621    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 07 10:14:21.858851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858661    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 07 10:14:21.858851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.858706    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xmqxd\""
Sep 07 10:14:21.861734 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.861712    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 07 10:14:21.864378 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.864359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 07 10:14:21.867594 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.867576    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:14:21.898470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.895432    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="511c2bb9-cc97-4f15-b6b9-2c25e5f621f4" path="/var/lib/kubelet/pods/511c2bb9-cc97-4f15-b6b9-2c25e5f621f4/volumes"
Sep 07 10:14:21.999937 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.999913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.999941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.999959    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:21.999975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-config\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000110    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000358    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cctq\" (UniqueName: \"kubernetes.io/projected/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-kube-api-access-4cctq\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000438    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000477    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.000596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.000502    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.101708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.101708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-config\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.101708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.101864 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101716    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.101864 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.101962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.101962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.101962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.101993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cctq\" (UniqueName: \"kubernetes.io/projected/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-kube-api-access-4cctq\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102131    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102680 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102666    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.102848 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.102820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.103211 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.103187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.103738 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.103712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.104700 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.104517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.104700 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.104521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.105364 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.104836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.105364 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.104908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.105364 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.105170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-config\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.105529 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.105456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.105529 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.105478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.106159 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.106130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.106275 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.106247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.106403 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.106377    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.107049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.107021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.107696 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.107669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.107772 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.107742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.107900 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.107878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.108375 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.108356    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.111491 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.111474    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cctq\" (UniqueName: \"kubernetes.io/projected/7f8a3db7-9b27-4511-881b-a09e8ffa9b6e-kube-api-access-4cctq\") pod \"prometheus-k8s-0\" (UID: \"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.165423 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.165401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:22.443278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.443254    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 07 10:14:22.444855 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:14:22.444822    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8a3db7_9b27_4511_881b_a09e8ffa9b6e.slice/crio-3885bbc185eb6a2f524369f5bebc616d73f9982004ac96d505071f7d64186378 WatchSource:0}: Error finding container 3885bbc185eb6a2f524369f5bebc616d73f9982004ac96d505071f7d64186378: Status 404 returned error can't find the container with id 3885bbc185eb6a2f524369f5bebc616d73f9982004ac96d505071f7d64186378
Sep 07 10:14:22.803243 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.803208    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f8a3db7-9b27-4511-881b-a09e8ffa9b6e" containerID="ee0ccdb8dbc2ba18edee0218f94ec7a6f1350ae1478838cca4b806902d64bf2a" exitCode=0
Sep 07 10:14:22.803383 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.803259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e","Type":"ContainerDied","Data":"ee0ccdb8dbc2ba18edee0218f94ec7a6f1350ae1478838cca4b806902d64bf2a"}
Sep 07 10:14:22.803383 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:22.803323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e","Type":"ContainerStarted","Data":"3885bbc185eb6a2f524369f5bebc616d73f9982004ac96d505071f7d64186378"}
Sep 07 10:14:23.811572 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:23.811536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e","Type":"ContainerStarted","Data":"365dfbb1d1fc3cdee41d5ea9d4b98d4ca72f347f595a31709efa3274d2c6aeee"}
Sep 07 10:14:23.811952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:23.811605    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e","Type":"ContainerStarted","Data":"2323577d704d1244b13d118a2e1489b066d252e396c542feed32da971952c043"}
Sep 07 10:14:23.811952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:23.811636    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e","Type":"ContainerStarted","Data":"e6e8972fb662ce6fae5fe5784056e4eba26834acb35da197a73dac80cabd7d52"}
Sep 07 10:14:23.811952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:23.811667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e","Type":"ContainerStarted","Data":"a66aefc1fe9b1b29bea66ee88a8cf3bd1de89a7fe18124f53ef27f0bd1593b54"}
Sep 07 10:14:23.811952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:23.811692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e","Type":"ContainerStarted","Data":"0c47e1cae54af138c7f298d8390818ce9f6206f89f7c5780d616fd231611cede"}
Sep 07 10:14:23.811952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:23.811714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f8a3db7-9b27-4511-881b-a09e8ffa9b6e","Type":"ContainerStarted","Data":"63eb395fd16c3289ae297c570297639bd3bbddf4abbc4a5da4526818a2fb88db"}
Sep 07 10:14:23.856943 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:23.856894    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8405761959999998 podStartE2EDuration="2.856878411s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.302215ms" isStatefulPod=false podCreationTimestamp="2026-09-07 10:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:14:23.855256985 +0000 UTC m=+170.403771973" watchObservedRunningTime="2026-09-07 10:14:23.856878411 +0000 UTC m=+170.405393397"
Sep 07 10:14:24.516911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:24.516877    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:24.570065 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:24.570029    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:27.165710 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:27.165634    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:14:31.111568 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:31.111529    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:31.163009 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:31.162963    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:35.046427 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:35.046381    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:35.145663 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:35.145633    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:35.203366 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:35.203332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:41.653082 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:41.653036    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76766cf945-v9bfw"]
Sep 07 10:14:41.765773 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:41.765739    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:41.864052 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:41.864016    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:41.927974 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:41.927902    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:57.747626 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:57.747580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:14:57.800463 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:14:57.800439    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:15:06.672818 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.672756    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76766cf945-v9bfw" podUID="97149142-5e27-4cc5-89b2-f35caea58ab8" containerName="console" containerID="cri-o://23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928" gracePeriod=15
Sep 07 10:15:06.904494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.904471    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:15:06.966732 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.966657    2560 generic.go:356] "Generic (PLEG): container finished" podID="97149142-5e27-4cc5-89b2-f35caea58ab8" containerID="23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928" exitCode=0
Sep 07 10:15:06.966732 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.966689    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76766cf945-v9bfw" event={"ID":"97149142-5e27-4cc5-89b2-f35caea58ab8","Type":"ContainerDied","Data":"23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928"}
Sep 07 10:15:06.966911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.966738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76766cf945-v9bfw" event={"ID":"97149142-5e27-4cc5-89b2-f35caea58ab8","Type":"ContainerDied","Data":"0ec1c9e9f77377b8c8f65a62a5df2f8cc88bd590f76f4313970c4a4546fbd088"}
Sep 07 10:15:06.966911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.966764    2560 scope.go:117] "RemoveContainer" containerID="23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928"
Sep 07 10:15:06.966911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.966702    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76766cf945-v9bfw"
Sep 07 10:15:06.975092 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.974901    2560 scope.go:117] "RemoveContainer" containerID="23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928"
Sep 07 10:15:06.976620 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:15:06.976600    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928\": container with ID starting with 23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928 not found: ID does not exist" containerID="23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928"
Sep 07 10:15:06.976709 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:06.976628    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928"} err="failed to get container status \"23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928\": rpc error: code = NotFound desc = could not find container \"23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928\": container with ID starting with 23a9fc1e0209985eaf2597969de19752d5ff48cfe1ca457e712a9897bb4a0928 not found: ID does not exist"
Sep 07 10:15:07.038017 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.037994    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-console-config\") pod \"97149142-5e27-4cc5-89b2-f35caea58ab8\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") "
Sep 07 10:15:07.038111 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038022    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-oauth-serving-cert\") pod \"97149142-5e27-4cc5-89b2-f35caea58ab8\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") "
Sep 07 10:15:07.038111 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm7d5\" (UniqueName: \"kubernetes.io/projected/97149142-5e27-4cc5-89b2-f35caea58ab8-kube-api-access-wm7d5\") pod \"97149142-5e27-4cc5-89b2-f35caea58ab8\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") "
Sep 07 10:15:07.038111 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-oauth-config\") pod \"97149142-5e27-4cc5-89b2-f35caea58ab8\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") "
Sep 07 10:15:07.038275 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038255    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-serving-cert\") pod \"97149142-5e27-4cc5-89b2-f35caea58ab8\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") "
Sep 07 10:15:07.038339 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-trusted-ca-bundle\") pod \"97149142-5e27-4cc5-89b2-f35caea58ab8\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") "
Sep 07 10:15:07.038447 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038425    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-service-ca\") pod \"97149142-5e27-4cc5-89b2-f35caea58ab8\" (UID: \"97149142-5e27-4cc5-89b2-f35caea58ab8\") "
Sep 07 10:15:07.038447 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038433    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97149142-5e27-4cc5-89b2-f35caea58ab8" (UID: "97149142-5e27-4cc5-89b2-f35caea58ab8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:15:07.038573 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038454    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-console-config" (OuterVolumeSpecName: "console-config") pod "97149142-5e27-4cc5-89b2-f35caea58ab8" (UID: "97149142-5e27-4cc5-89b2-f35caea58ab8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:15:07.038686 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038660    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-console-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:15:07.038756 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038711    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-oauth-serving-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:15:07.038834 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038805    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "97149142-5e27-4cc5-89b2-f35caea58ab8" (UID: "97149142-5e27-4cc5-89b2-f35caea58ab8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:15:07.038899 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.038833    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-service-ca" (OuterVolumeSpecName: "service-ca") pod "97149142-5e27-4cc5-89b2-f35caea58ab8" (UID: "97149142-5e27-4cc5-89b2-f35caea58ab8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:15:07.040146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.040123    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97149142-5e27-4cc5-89b2-f35caea58ab8" (UID: "97149142-5e27-4cc5-89b2-f35caea58ab8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:15:07.040226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.040199    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97149142-5e27-4cc5-89b2-f35caea58ab8-kube-api-access-wm7d5" (OuterVolumeSpecName: "kube-api-access-wm7d5") pod "97149142-5e27-4cc5-89b2-f35caea58ab8" (UID: "97149142-5e27-4cc5-89b2-f35caea58ab8"). InnerVolumeSpecName "kube-api-access-wm7d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:15:07.040265 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.040241    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97149142-5e27-4cc5-89b2-f35caea58ab8" (UID: "97149142-5e27-4cc5-89b2-f35caea58ab8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:15:07.139389 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.139355    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-oauth-config\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:15:07.139389 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.139382    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97149142-5e27-4cc5-89b2-f35caea58ab8-console-serving-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:15:07.139389 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.139391    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-trusted-ca-bundle\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:15:07.139594 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.139400    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97149142-5e27-4cc5-89b2-f35caea58ab8-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:15:07.139594 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.139408    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm7d5\" (UniqueName: \"kubernetes.io/projected/97149142-5e27-4cc5-89b2-f35caea58ab8-kube-api-access-wm7d5\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:15:07.287290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.287267    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76766cf945-v9bfw"]
Sep 07 10:15:07.289381 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.289358    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76766cf945-v9bfw"]
Sep 07 10:15:07.894193 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:07.894161    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97149142-5e27-4cc5-89b2-f35caea58ab8" path="/var/lib/kubelet/pods/97149142-5e27-4cc5-89b2-f35caea58ab8/volumes"
Sep 07 10:15:22.166221 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:22.166181    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:15:22.181732 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:22.181707    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:15:23.044477 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:23.044445    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 07 10:15:32.447212 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:32.447167    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:15:32.538638 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:32.538610    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:15:54.067665 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:54.067633    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:15:54.857708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:54.857667    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:15:57.687444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:57.687357    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:15:57.814686 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:57.814655    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:15:59.990925 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:15:59.990872    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:00.086297 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:00.086263    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:00.505033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:00.504995    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:00.611961 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:00.611931    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:02.198213 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:02.198177    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:02.803837 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:02.803790    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:04.470325 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:04.470284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:04.529748 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:04.529716    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:09.390494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:09.390456    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:09.491595 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:09.491561    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:10.812206 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:10.812172    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:24.489693 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:24.489651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:24.588385 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:24.588353    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:32.949966 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:32.949935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:33.776889 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:33.776862    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 10:16:35.821581 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:35.821546    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:37.798811 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:37.798775    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:49.174841 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.174812    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:49.222768 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.222741    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"]
Sep 07 10:16:49.223048 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.223036    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97149142-5e27-4cc5-89b2-f35caea58ab8" containerName="console"
Sep 07 10:16:49.223115 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.223049    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="97149142-5e27-4cc5-89b2-f35caea58ab8" containerName="console"
Sep 07 10:16:49.223115 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.223113    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="97149142-5e27-4cc5-89b2-f35caea58ab8" containerName="console"
Sep 07 10:16:49.225912 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.225894    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.228490 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.228466    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 07 10:16:49.228695 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.228681    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 07 10:16:49.229480 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.229463    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 10:16:49.229697 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.229682    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 10:16:49.229807 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.229787    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-w9qqf\""
Sep 07 10:16:49.229852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.229838    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 07 10:16:49.239104 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.239083    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"]
Sep 07 10:16:49.283385 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.283366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3de56e5-6711-4537-b795-4660f5c8dde5-manager-config\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.283501 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.283423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3de56e5-6711-4537-b795-4660f5c8dde5-metrics-cert\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.283567 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.283498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q622k\" (UniqueName: \"kubernetes.io/projected/b3de56e5-6711-4537-b795-4660f5c8dde5-kube-api-access-q622k\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.283567 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.283533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3de56e5-6711-4537-b795-4660f5c8dde5-cert\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.337107 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.337082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:49.384158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.384132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3de56e5-6711-4537-b795-4660f5c8dde5-manager-config\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.384274 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.384195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3de56e5-6711-4537-b795-4660f5c8dde5-metrics-cert\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.384274 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.384232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q622k\" (UniqueName: \"kubernetes.io/projected/b3de56e5-6711-4537-b795-4660f5c8dde5-kube-api-access-q622k\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.384274 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.384264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3de56e5-6711-4537-b795-4660f5c8dde5-cert\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.384822 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.384798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b3de56e5-6711-4537-b795-4660f5c8dde5-manager-config\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.386629 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.386608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3de56e5-6711-4537-b795-4660f5c8dde5-metrics-cert\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.386711 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.386676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3de56e5-6711-4537-b795-4660f5c8dde5-cert\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.399672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.399645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q622k\" (UniqueName: \"kubernetes.io/projected/b3de56e5-6711-4537-b795-4660f5c8dde5-kube-api-access-q622k\") pod \"lws-controller-manager-6c54b9f868-p2zsj\" (UID: \"b3de56e5-6711-4537-b795-4660f5c8dde5\") " pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.534532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.534509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:49.803465 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.803442    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"]
Sep 07 10:16:49.805447 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:16:49.805424    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3de56e5_6711_4537_b795_4660f5c8dde5.slice/crio-6b5bfae897609212e2780690e853de03994fafefa250a82f83ace8959ab28e91 WatchSource:0}: Error finding container 6b5bfae897609212e2780690e853de03994fafefa250a82f83ace8959ab28e91: Status 404 returned error can't find the container with id 6b5bfae897609212e2780690e853de03994fafefa250a82f83ace8959ab28e91
Sep 07 10:16:49.807416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:49.807400    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:16:50.348255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:50.348223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj" event={"ID":"b3de56e5-6711-4537-b795-4660f5c8dde5","Type":"ContainerStarted","Data":"6b5bfae897609212e2780690e853de03994fafefa250a82f83ace8959ab28e91"}
Sep 07 10:16:53.361213 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:53.361176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj" event={"ID":"b3de56e5-6711-4537-b795-4660f5c8dde5","Type":"ContainerStarted","Data":"cad684fe5bd1832b8c3146c5488549910e49cf16a0ad691aabc6fc2604fe6c56"}
Sep 07 10:16:53.361648 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:53.361240    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:16:53.396158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:53.396109    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj" podStartSLOduration=1.837375884 podStartE2EDuration="4.396091933s" totalImagesPullingTime="2.558716049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:16:53.395533459 +0000 UTC m=+319.944048451" watchObservedRunningTime="2026-09-07 10:16:53.396091933 +0000 UTC m=+319.944606919"
Sep 07 10:16:55.091290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:55.091260    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:55.157885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:55.157858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:55.907853 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:55.907821    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:56.125469 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.125439    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:56.225274 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.225208    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:56.279807 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.279783    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"]
Sep 07 10:16:56.283924 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.283904    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.286565 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.286544    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 10:16:56.286893 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.286876    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 10:16:56.287118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.287104    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q77qk\""
Sep 07 10:16:56.287229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.287214    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 10:16:56.287432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.287414    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 10:16:56.297374 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.297355    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"]
Sep 07 10:16:56.337241 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.337218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/864c4745-6b6f-4b85-bd67-959a99ead247-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.337356 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.337245    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/864c4745-6b6f-4b85-bd67-959a99ead247-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.337356 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.337277    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/864c4745-6b6f-4b85-bd67-959a99ead247-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.337470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.337388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxkw\" (UniqueName: \"kubernetes.io/projected/864c4745-6b6f-4b85-bd67-959a99ead247-kube-api-access-6cxkw\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.337470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.337426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/864c4745-6b6f-4b85-bd67-959a99ead247-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.437976 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.437947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/864c4745-6b6f-4b85-bd67-959a99ead247-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.438166 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.437993    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxkw\" (UniqueName: \"kubernetes.io/projected/864c4745-6b6f-4b85-bd67-959a99ead247-kube-api-access-6cxkw\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.438166 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.438015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/864c4745-6b6f-4b85-bd67-959a99ead247-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.438166 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.438045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/864c4745-6b6f-4b85-bd67-959a99ead247-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.438346 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.438206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/864c4745-6b6f-4b85-bd67-959a99ead247-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.438403 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.438389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/864c4745-6b6f-4b85-bd67-959a99ead247-manifests\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.440391 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.440361    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/864c4745-6b6f-4b85-bd67-959a99ead247-tmp\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.440602 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.440583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/864c4745-6b6f-4b85-bd67-959a99ead247-webhook-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.440737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.440716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/864c4745-6b6f-4b85-bd67-959a99ead247-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.446891 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.446865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxkw\" (UniqueName: \"kubernetes.io/projected/864c4745-6b6f-4b85-bd67-959a99ead247-kube-api-access-6cxkw\") pod \"opendatahub-operator-controller-manager-685c7f779d-gvgn5\" (UID: \"864c4745-6b6f-4b85-bd67-959a99ead247\") " pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.454238 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.454209    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:56.559264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.559222    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:56.594032 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.594000    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:16:56.657979 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.657916    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:56.756255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.756222    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:56.848564 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.848497    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:56.870413 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.870392    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"]
Sep 07 10:16:56.873008 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:16:56.872983    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864c4745_6b6f_4b85_bd67_959a99ead247.slice/crio-bba26a8b6c29010acd0c5e36277b159e159a1297e126f0cc99efc63b14316351 WatchSource:0}: Error finding container bba26a8b6c29010acd0c5e36277b159e159a1297e126f0cc99efc63b14316351: Status 404 returned error can't find the container with id bba26a8b6c29010acd0c5e36277b159e159a1297e126f0cc99efc63b14316351
Sep 07 10:16:56.924717 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:56.924681    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:57.017939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:57.017912    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:16:57.377356 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:16:57.377324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5" event={"ID":"864c4745-6b6f-4b85-bd67-959a99ead247","Type":"ContainerStarted","Data":"bba26a8b6c29010acd0c5e36277b159e159a1297e126f0cc99efc63b14316351"}
Sep 07 10:17:00.391539 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:00.391500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5" event={"ID":"864c4745-6b6f-4b85-bd67-959a99ead247","Type":"ContainerStarted","Data":"2ec1379f895f0dfa53b84c39357561a0dfa29f542218f1b54b616681c36822ce"}
Sep 07 10:17:01.396928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:01.396889    2560 generic.go:356] "Generic (PLEG): container finished" podID="864c4745-6b6f-4b85-bd67-959a99ead247" containerID="2ec1379f895f0dfa53b84c39357561a0dfa29f542218f1b54b616681c36822ce" exitCode=0
Sep 07 10:17:01.397315 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:01.396968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5" event={"ID":"864c4745-6b6f-4b85-bd67-959a99ead247","Type":"ContainerDied","Data":"2ec1379f895f0dfa53b84c39357561a0dfa29f542218f1b54b616681c36822ce"}
Sep 07 10:17:02.402194 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:02.402156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5" event={"ID":"864c4745-6b6f-4b85-bd67-959a99ead247","Type":"ContainerStarted","Data":"1d958139fd01116d6584040f48f472349e8dda106c0b56d8314eae7e1a52f9dd"}
Sep 07 10:17:02.402566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:02.402299    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:17:02.425235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:02.425185    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5" podStartSLOduration=2.105353699 podStartE2EDuration="6.425168957s" totalImagesPullingTime="3.31471068s" totalInitContainerRuntime="1.005104578s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:16:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:02.42290061 +0000 UTC m=+328.971415611" watchObservedRunningTime="2026-09-07 10:17:02.425168957 +0000 UTC m=+328.973683943"
Sep 07 10:17:04.368053 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:04.368024    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c54b9f868-p2zsj"
Sep 07 10:17:05.000047 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:05.000007    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:08.089826 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:08.089793    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:08.202762 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:08.202720    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:13.408299 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:13.408266    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-685c7f779d-gvgn5"
Sep 07 10:17:15.129466 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:15.129424    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:15.239467 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:15.239428    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:16.863908 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:16.863874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:16.925387 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:16.925360    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"]
Sep 07 10:17:17.050540 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.050507    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"]
Sep 07 10:17:17.050687 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.050613    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.053046 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.053028    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 10:17:17.053279 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.053216    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 10:17:17.053356 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.053283    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 10:17:17.054029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.054009    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 10:17:17.054147 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.054039    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zr766\""
Sep 07 10:17:17.112572 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.112544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-tmp\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.112697 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.112590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-tls-certs\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.112697 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.112615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-542h2\" (UniqueName: \"kubernetes.io/projected/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-kube-api-access-542h2\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.213457 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.213394    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-tmp\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.213457 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.213436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-tls-certs\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.213615 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.213459    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542h2\" (UniqueName: \"kubernetes.io/projected/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-kube-api-access-542h2\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.215456 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.215431    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-tmp\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.215757 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.215737    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-tls-certs\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.222137 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.222112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-542h2\" (UniqueName: \"kubernetes.io/projected/64dd96ce-65de-48bc-ab25-9d0e2e7e36a4-kube-api-access-542h2\") pod \"kube-auth-proxy-7959f5887b-bgzv5\" (UID: \"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4\") " pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.297119 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.297065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:17.359611 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.359573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"
Sep 07 10:17:17.649710 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:17.649679    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5"]
Sep 07 10:17:17.653221 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:17:17.653190    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dd96ce_65de_48bc_ab25_9d0e2e7e36a4.slice/crio-cafc43b3e64da83b6c0a7e1144d2f8081adbd4457932ba2ae2cb6dc8a4557932 WatchSource:0}: Error finding container cafc43b3e64da83b6c0a7e1144d2f8081adbd4457932ba2ae2cb6dc8a4557932: Status 404 returned error can't find the container with id cafc43b3e64da83b6c0a7e1144d2f8081adbd4457932ba2ae2cb6dc8a4557932
Sep 07 10:17:18.012016 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:18.011934    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:18.105388 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:18.105355    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:18.465952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:18.465913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5" event={"ID":"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4","Type":"ContainerStarted","Data":"cafc43b3e64da83b6c0a7e1144d2f8081adbd4457932ba2ae2cb6dc8a4557932"}
Sep 07 10:17:18.893906 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:18.893859    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:21.479752 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:21.479712    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5" event={"ID":"64dd96ce-65de-48bc-ab25-9d0e2e7e36a4","Type":"ContainerStarted","Data":"4db19d1766aa931c86871c0bded835d5fc1ddff3ba7d9d5635f6eb9a45aa0b0c"}
Sep 07 10:17:21.495126 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:21.495057    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7959f5887b-bgzv5" podStartSLOduration=2.656663332 podStartE2EDuration="5.495041418s" totalImagesPullingTime="2.838378086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:21.494195453 +0000 UTC m=+348.042710442" watchObservedRunningTime="2026-09-07 10:17:21.495041418 +0000 UTC m=+348.043556404"
Sep 07 10:17:23.147022 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:23.146990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:23.248119 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:23.248057    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:23.341710 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:23.341676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:23.410999 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:23.410925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:23.659606 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:23.659566    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:24.194555 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:24.194518    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:24.295393 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:24.295361    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:25.171905 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:25.171869    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:25.225167 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:25.225133    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:26.309018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:26.308939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:26.408296 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:26.408263    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:26.508990 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:26.508948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:26.819823 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:26.819789    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:26.931463 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:26.931434    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:36.428583 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:36.428543    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:36.577230 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:36.577193    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:36.687456 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:36.687382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:37.811873 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:37.811834    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:37.911183 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:37.911155    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:37.969794 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:37.969768    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:41.805017 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.804988    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b47fc8985-fb67l"]
Sep 07 10:17:41.809779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.809761    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:41.813817 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.813794    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 10:17:41.814370 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.814353    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 10:17:41.814524 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.814509    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ndkq9\""
Sep 07 10:17:41.814612 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.814598    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 10:17:41.814701 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.814686    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 10:17:41.814783 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.814765    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 10:17:41.814913 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.814864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 10:17:41.815140 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.815124    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 10:17:41.824675 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.824654    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b47fc8985-fb67l"]
Sep 07 10:17:41.826169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.826147    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 10:17:41.912487 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.912450    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:41.915511 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.915489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffv85\" (UniqueName: \"kubernetes.io/projected/97751574-c5d9-4128-a817-5829988af14e-kube-api-access-ffv85\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:41.915585 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.915522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97751574-c5d9-4128-a817-5829988af14e-console-oauth-config\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:41.915585 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.915543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-service-ca\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:41.915585 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.915570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-trusted-ca-bundle\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:41.915684 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.915603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-oauth-serving-cert\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:41.915684 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.915641    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97751574-c5d9-4128-a817-5829988af14e-console-serving-cert\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:41.915684 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.915658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-console-config\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:41.991096 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:41.991056    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:42.016081 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.016049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97751574-c5d9-4128-a817-5829988af14e-console-serving-cert\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.016174 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.016091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-console-config\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.016324 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.016291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffv85\" (UniqueName: \"kubernetes.io/projected/97751574-c5d9-4128-a817-5829988af14e-kube-api-access-ffv85\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.016458 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.016331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97751574-c5d9-4128-a817-5829988af14e-console-oauth-config\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.016458 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.016378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-service-ca\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.016458 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.016404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-trusted-ca-bundle\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.016458 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.016440    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-oauth-serving-cert\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.016857 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.016837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-console-config\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.017098 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.017061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-service-ca\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.017249 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.017225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-oauth-serving-cert\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.017886 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.017866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97751574-c5d9-4128-a817-5829988af14e-trusted-ca-bundle\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.018656 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.018634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97751574-c5d9-4128-a817-5829988af14e-console-serving-cert\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.018964 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.018946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97751574-c5d9-4128-a817-5829988af14e-console-oauth-config\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.024988 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.024961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffv85\" (UniqueName: \"kubernetes.io/projected/97751574-c5d9-4128-a817-5829988af14e-kube-api-access-ffv85\") pod \"console-b47fc8985-fb67l\" (UID: \"97751574-c5d9-4128-a817-5829988af14e\") " pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.091584 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.091491    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:42.118156 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.118118    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:42.388672 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.388646    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b47fc8985-fb67l"]
Sep 07 10:17:42.391261 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:17:42.391229    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97751574_c5d9_4128_a817_5829988af14e.slice/crio-64562d49841169da20d2d8300b06721a7dcf3f642e84b4013b8759e79913ed2a WatchSource:0}: Error finding container 64562d49841169da20d2d8300b06721a7dcf3f642e84b4013b8759e79913ed2a: Status 404 returned error can't find the container with id 64562d49841169da20d2d8300b06721a7dcf3f642e84b4013b8759e79913ed2a
Sep 07 10:17:42.563081 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.563044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b47fc8985-fb67l" event={"ID":"97751574-c5d9-4128-a817-5829988af14e","Type":"ContainerStarted","Data":"f6be4af427a277d7267bfa0b4fe99d1352c0b0d2212df8959117ae011e20dc53"}
Sep 07 10:17:42.563243 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.563110    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b47fc8985-fb67l" event={"ID":"97751574-c5d9-4128-a817-5829988af14e","Type":"ContainerStarted","Data":"64562d49841169da20d2d8300b06721a7dcf3f642e84b4013b8759e79913ed2a"}
Sep 07 10:17:42.582157 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.582112    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b47fc8985-fb67l" podStartSLOduration=1.582092409 podStartE2EDuration="1.582092409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:17:42.581367254 +0000 UTC m=+369.129882240" watchObservedRunningTime="2026-09-07 10:17:42.582092409 +0000 UTC m=+369.130607387"
Sep 07 10:17:42.587660 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.587623    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:42.644360 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:42.644289    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:52.119195 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:52.119145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:52.119691 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:52.119231    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:52.123873 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:52.123852    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:52.607289 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:52.607260    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b47fc8985-fb67l"
Sep 07 10:17:52.750288 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:52.750248    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:52.850730 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:52.850691    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:17:52.952481 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:17:52.952392    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:18.756389 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:18.756356    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:27.267723 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:27.267689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:27.375941 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:27.375907    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:40.227357 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:40.227320    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:43.027641 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:43.027588    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:43.870211 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:43.870174    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:44.597771 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:44.597726    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:44.996516 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:44.996433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:45.257819 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:45.257726    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:46.634773 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:46.634737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:47.631859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:47.631822    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:47.734149 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:47.734120    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:48.006887 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:48.006799    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:48.109572 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:48.109532    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:48.404710 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:48.404666    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:48.506587 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:48.506548    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:49.037683 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:49.037647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:49.148715 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:49.148670    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:52.806496 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:52.806461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:52.911122 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:52.911085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:55.805592 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:55.805550    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:55.906087 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:55.906045    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:58.676022 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.675948    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"]
Sep 07 10:18:58.679086 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.679056    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.679874 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.679845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:58.681501 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.681482    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 10:18:58.681580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.681505    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5pfgz\""
Sep 07 10:18:58.681580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.681529    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 07 10:18:58.681580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.681482    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 07 10:18:58.682414 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.682399    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 10:18:58.686830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.686812    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"]
Sep 07 10:18:58.730319 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.730282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/86b2146f-98e9-4811-85ad-a4684ba61ed1-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.730455 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.730326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86b2146f-98e9-4811-85ad-a4684ba61ed1-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.730455 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.730375    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzb4k\" (UniqueName: \"kubernetes.io/projected/86b2146f-98e9-4811-85ad-a4684ba61ed1-kube-api-access-kzb4k\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.779669 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.779640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:18:58.831476 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.831447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/86b2146f-98e9-4811-85ad-a4684ba61ed1-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.831620 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.831484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86b2146f-98e9-4811-85ad-a4684ba61ed1-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.831620 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.831503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzb4k\" (UniqueName: \"kubernetes.io/projected/86b2146f-98e9-4811-85ad-a4684ba61ed1-kube-api-access-kzb4k\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.832109 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.832089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/86b2146f-98e9-4811-85ad-a4684ba61ed1-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.833922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.833900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/86b2146f-98e9-4811-85ad-a4684ba61ed1-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.840622 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.840602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzb4k\" (UniqueName: \"kubernetes.io/projected/86b2146f-98e9-4811-85ad-a4684ba61ed1-kube-api-access-kzb4k\") pod \"kuadrant-console-plugin-598f86468b-g4kf6\" (UID: \"86b2146f-98e9-4811-85ad-a4684ba61ed1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:58.989696 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:58.989616    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"
Sep 07 10:18:59.262579 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:59.262424    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6"]
Sep 07 10:18:59.265280 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:18:59.265253    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86b2146f_98e9_4811_85ad_a4684ba61ed1.slice/crio-8ceaf41cb199cc0bc0a1ee50a36ab304dd3068661608d7cb5f0798171349011a WatchSource:0}: Error finding container 8ceaf41cb199cc0bc0a1ee50a36ab304dd3068661608d7cb5f0798171349011a: Status 404 returned error can't find the container with id 8ceaf41cb199cc0bc0a1ee50a36ab304dd3068661608d7cb5f0798171349011a
Sep 07 10:18:59.873147 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:18:59.873111    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6" event={"ID":"86b2146f-98e9-4811-85ad-a4684ba61ed1","Type":"ContainerStarted","Data":"8ceaf41cb199cc0bc0a1ee50a36ab304dd3068661608d7cb5f0798171349011a"}
Sep 07 10:19:01.530160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:01.530108    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:01.626686 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:01.626598    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:03.894761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:03.894730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6" event={"ID":"86b2146f-98e9-4811-85ad-a4684ba61ed1","Type":"ContainerStarted","Data":"c44aa6b91ef52ce3d774a8e93c8aaaf84dbce0cada5d8d51ea0aa2fe94f02db3"}
Sep 07 10:19:03.912415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:03.912368    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-g4kf6" podStartSLOduration=1.423793745 podStartE2EDuration="5.912349021s" totalImagesPullingTime="4.488555276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:03.911926312 +0000 UTC m=+450.460441299" watchObservedRunningTime="2026-09-07 10:19:03.912349021 +0000 UTC m=+450.460864008"
Sep 07 10:19:06.152399 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:06.152369    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:06.253228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:06.253191    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:06.352698 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:06.352657    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:12.827040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:12.827003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:12.966822 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:12.966784    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:13.866361 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:13.866315    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:15.800613 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:15.800572    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:15.901257 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:15.901226    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:21.601980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:21.601941    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:21.657059 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:21.657028    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:22.236498 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:22.236468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:22.335263 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:22.335223    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:25.680442 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:25.680401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:25.778414 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:25.778378    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:25.882348 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:25.882309    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:30.828061 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:30.828019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:30.928265 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:30.928235    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:38.560047 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:38.560014    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:38.962353 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:38.962264    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:52.869341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:52.869301    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:52.970498 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:52.970462    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:53.031061 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.031030    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:53.253281 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.253196    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:19:53.256488 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.256472    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:53.258729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.258707    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 07 10:19:53.263530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.263506    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:19:53.278083 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.278050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqhl\" (UniqueName: \"kubernetes.io/projected/1d95000e-1fb2-4b8a-9a2f-da48e0eefccc-kube-api-access-cxqhl\") pod \"limitador-limitador-85d9db4d5d-mqmqd\" (UID: \"1d95000e-1fb2-4b8a-9a2f-da48e0eefccc\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:53.278190 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.278107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d95000e-1fb2-4b8a-9a2f-da48e0eefccc-config-file\") pod \"limitador-limitador-85d9db4d5d-mqmqd\" (UID: \"1d95000e-1fb2-4b8a-9a2f-da48e0eefccc\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:53.289634 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.289609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:19:53.378792 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.378765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqhl\" (UniqueName: \"kubernetes.io/projected/1d95000e-1fb2-4b8a-9a2f-da48e0eefccc-kube-api-access-cxqhl\") pod \"limitador-limitador-85d9db4d5d-mqmqd\" (UID: \"1d95000e-1fb2-4b8a-9a2f-da48e0eefccc\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:53.378908 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.378803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d95000e-1fb2-4b8a-9a2f-da48e0eefccc-config-file\") pod \"limitador-limitador-85d9db4d5d-mqmqd\" (UID: \"1d95000e-1fb2-4b8a-9a2f-da48e0eefccc\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:53.379455 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.379437    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1d95000e-1fb2-4b8a-9a2f-da48e0eefccc-config-file\") pod \"limitador-limitador-85d9db4d5d-mqmqd\" (UID: \"1d95000e-1fb2-4b8a-9a2f-da48e0eefccc\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:53.388726 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.388702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqhl\" (UniqueName: \"kubernetes.io/projected/1d95000e-1fb2-4b8a-9a2f-da48e0eefccc-kube-api-access-cxqhl\") pod \"limitador-limitador-85d9db4d5d-mqmqd\" (UID: \"1d95000e-1fb2-4b8a-9a2f-da48e0eefccc\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:53.567256 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.567229    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:53.693551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.693523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:53.761146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.761119    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:19:53.764575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.764551    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bc6pj"]
Sep 07 10:19:53.767875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.767859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-bc6pj"
Sep 07 10:19:53.770350 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.770327    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cbrkq\""
Sep 07 10:19:53.774416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.774394    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bc6pj"]
Sep 07 10:19:53.781764 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.781650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7fj\" (UniqueName: \"kubernetes.io/projected/0db54ffa-9853-4473-b568-63227b6a0622-kube-api-access-hj7fj\") pod \"authorino-c548b5d7d-bc6pj\" (UID: \"0db54ffa-9853-4473-b568-63227b6a0622\") " pod="kuadrant-system/authorino-c548b5d7d-bc6pj"
Sep 07 10:19:53.836899 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.836872    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:19:53.839163 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:19:53.839132    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d95000e_1fb2_4b8a_9a2f_da48e0eefccc.slice/crio-ed94223260f29b7a2afdf271107727fdfcb5ff048fb96f6b26d461d24d49d023 WatchSource:0}: Error finding container ed94223260f29b7a2afdf271107727fdfcb5ff048fb96f6b26d461d24d49d023: Status 404 returned error can't find the container with id ed94223260f29b7a2afdf271107727fdfcb5ff048fb96f6b26d461d24d49d023
Sep 07 10:19:53.882897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.882852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7fj\" (UniqueName: \"kubernetes.io/projected/0db54ffa-9853-4473-b568-63227b6a0622-kube-api-access-hj7fj\") pod \"authorino-c548b5d7d-bc6pj\" (UID: \"0db54ffa-9853-4473-b568-63227b6a0622\") " pod="kuadrant-system/authorino-c548b5d7d-bc6pj"
Sep 07 10:19:53.891498 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:53.891471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7fj\" (UniqueName: \"kubernetes.io/projected/0db54ffa-9853-4473-b568-63227b6a0622-kube-api-access-hj7fj\") pod \"authorino-c548b5d7d-bc6pj\" (UID: \"0db54ffa-9853-4473-b568-63227b6a0622\") " pod="kuadrant-system/authorino-c548b5d7d-bc6pj"
Sep 07 10:19:54.078586 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:54.078553    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-bc6pj"
Sep 07 10:19:54.091314 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:54.091282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd" event={"ID":"1d95000e-1fb2-4b8a-9a2f-da48e0eefccc","Type":"ContainerStarted","Data":"ed94223260f29b7a2afdf271107727fdfcb5ff048fb96f6b26d461d24d49d023"}
Sep 07 10:19:54.345709 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:54.345631    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bc6pj"]
Sep 07 10:19:54.348318 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:19:54.348290    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0db54ffa_9853_4473_b568_63227b6a0622.slice/crio-ebcfa258918b04ac7e24669bf1b78cd34dd448b61e944249c86687413b507977 WatchSource:0}: Error finding container ebcfa258918b04ac7e24669bf1b78cd34dd448b61e944249c86687413b507977: Status 404 returned error can't find the container with id ebcfa258918b04ac7e24669bf1b78cd34dd448b61e944249c86687413b507977
Sep 07 10:19:55.098792 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:55.098746    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-bc6pj" event={"ID":"0db54ffa-9853-4473-b568-63227b6a0622","Type":"ContainerStarted","Data":"ebcfa258918b04ac7e24669bf1b78cd34dd448b61e944249c86687413b507977"}
Sep 07 10:19:57.108036 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:57.107929    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-bc6pj" event={"ID":"0db54ffa-9853-4473-b568-63227b6a0622","Type":"ContainerStarted","Data":"7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9"}
Sep 07 10:19:57.109464 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:57.109440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd" event={"ID":"1d95000e-1fb2-4b8a-9a2f-da48e0eefccc","Type":"ContainerStarted","Data":"c2480920fe3879e7de39120c7ccae20964c8db8b6b9e0c9b651e9023dfd77c94"}
Sep 07 10:19:57.109582 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:57.109501    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:19:57.122334 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:57.122288    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-bc6pj" podStartSLOduration=1.6742869950000001 podStartE2EDuration="4.122273169s" totalImagesPullingTime="2.447986174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:57.121206323 +0000 UTC m=+503.669721311" watchObservedRunningTime="2026-09-07 10:19:57.122273169 +0000 UTC m=+503.670788156"
Sep 07 10:19:57.137408 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:19:57.137365    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd" podStartSLOduration=1.850052241 podStartE2EDuration="4.137350868s" totalImagesPullingTime="2.287298627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:19:57.135839609 +0000 UTC m=+503.684354599" watchObservedRunningTime="2026-09-07 10:19:57.137350868 +0000 UTC m=+503.685865854"
Sep 07 10:20:08.114685 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:08.114650    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"
Sep 07 10:20:12.650158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:12.650118    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:12.803165 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:12.803137    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:13.104322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:13.104288    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:25.698393 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:25.698349    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:25.914446 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:25.914403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:26.206690 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.206660    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78b89f7fd9-tqsrl"]
Sep 07 10:20:26.209969 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.209952    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:20:26.212093 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.212056    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 07 10:20:26.215953 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.215932    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78b89f7fd9-tqsrl"]
Sep 07 10:20:26.337748 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.337682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/934a6d8d-2653-4e4c-8ffa-d5641995eec6-tls-cert\") pod \"authorino-78b89f7fd9-tqsrl\" (UID: \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\") " pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:20:26.337868 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.337749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xj46\" (UniqueName: \"kubernetes.io/projected/934a6d8d-2653-4e4c-8ffa-d5641995eec6-kube-api-access-6xj46\") pod \"authorino-78b89f7fd9-tqsrl\" (UID: \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\") " pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:20:26.438926 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.438888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xj46\" (UniqueName: \"kubernetes.io/projected/934a6d8d-2653-4e4c-8ffa-d5641995eec6-kube-api-access-6xj46\") pod \"authorino-78b89f7fd9-tqsrl\" (UID: \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\") " pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:20:26.439038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.438941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/934a6d8d-2653-4e4c-8ffa-d5641995eec6-tls-cert\") pod \"authorino-78b89f7fd9-tqsrl\" (UID: \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\") " pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:20:26.441415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.441395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/934a6d8d-2653-4e4c-8ffa-d5641995eec6-tls-cert\") pod \"authorino-78b89f7fd9-tqsrl\" (UID: \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\") " pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:20:26.448040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.448017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xj46\" (UniqueName: \"kubernetes.io/projected/934a6d8d-2653-4e4c-8ffa-d5641995eec6-kube-api-access-6xj46\") pod \"authorino-78b89f7fd9-tqsrl\" (UID: \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\") " pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:20:26.518799 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.518746    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:20:26.787283 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:26.787252    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78b89f7fd9-tqsrl"]
Sep 07 10:20:26.790627 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:20:26.790598    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod934a6d8d_2653_4e4c_8ffa_d5641995eec6.slice/crio-9c9751e66e4591d067a5610f45b449467299741145021ce077690316c87c53dd WatchSource:0}: Error finding container 9c9751e66e4591d067a5610f45b449467299741145021ce077690316c87c53dd: Status 404 returned error can't find the container with id 9c9751e66e4591d067a5610f45b449467299741145021ce077690316c87c53dd
Sep 07 10:20:27.232689 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:27.232612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl" event={"ID":"934a6d8d-2653-4e4c-8ffa-d5641995eec6","Type":"ContainerStarted","Data":"9c9751e66e4591d067a5610f45b449467299741145021ce077690316c87c53dd"}
Sep 07 10:20:28.238530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:28.238491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl" event={"ID":"934a6d8d-2653-4e4c-8ffa-d5641995eec6","Type":"ContainerStarted","Data":"15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca"}
Sep 07 10:20:28.253316 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:28.253271    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl" podStartSLOduration=1.6224647559999998 podStartE2EDuration="2.253258267s" totalImagesPullingTime="630.793511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:20:28.252514287 +0000 UTC m=+534.801029273" watchObservedRunningTime="2026-09-07 10:20:28.253258267 +0000 UTC m=+534.801773254"
Sep 07 10:20:28.292578 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:28.292549    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bc6pj"]
Sep 07 10:20:28.292743 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:28.292723    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-bc6pj" podUID="0db54ffa-9853-4473-b568-63227b6a0622" containerName="authorino" containerID="cri-o://7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9" gracePeriod=30
Sep 07 10:20:28.542981 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:28.542960    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-bc6pj"
Sep 07 10:20:28.657886 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:28.657857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj7fj\" (UniqueName: \"kubernetes.io/projected/0db54ffa-9853-4473-b568-63227b6a0622-kube-api-access-hj7fj\") pod \"0db54ffa-9853-4473-b568-63227b6a0622\" (UID: \"0db54ffa-9853-4473-b568-63227b6a0622\") "
Sep 07 10:20:28.659938 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:28.659911    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db54ffa-9853-4473-b568-63227b6a0622-kube-api-access-hj7fj" (OuterVolumeSpecName: "kube-api-access-hj7fj") pod "0db54ffa-9853-4473-b568-63227b6a0622" (UID: "0db54ffa-9853-4473-b568-63227b6a0622"). InnerVolumeSpecName "kube-api-access-hj7fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:20:28.758729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:28.758702    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7fj\" (UniqueName: \"kubernetes.io/projected/0db54ffa-9853-4473-b568-63227b6a0622-kube-api-access-hj7fj\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:20:29.244689 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.244649    2560 generic.go:356] "Generic (PLEG): container finished" podID="0db54ffa-9853-4473-b568-63227b6a0622" containerID="7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9" exitCode=0
Sep 07 10:20:29.245171 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.244695    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-bc6pj"
Sep 07 10:20:29.245171 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.244730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-bc6pj" event={"ID":"0db54ffa-9853-4473-b568-63227b6a0622","Type":"ContainerDied","Data":"7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9"}
Sep 07 10:20:29.245171 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.244775    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-bc6pj" event={"ID":"0db54ffa-9853-4473-b568-63227b6a0622","Type":"ContainerDied","Data":"ebcfa258918b04ac7e24669bf1b78cd34dd448b61e944249c86687413b507977"}
Sep 07 10:20:29.245171 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.244791    2560 scope.go:117] "RemoveContainer" containerID="7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9"
Sep 07 10:20:29.253291 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.253274    2560 scope.go:117] "RemoveContainer" containerID="7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9"
Sep 07 10:20:29.253532 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:20:29.253516    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9\": container with ID starting with 7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9 not found: ID does not exist" containerID="7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9"
Sep 07 10:20:29.253571 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.253540    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9"} err="failed to get container status \"7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9\": rpc error: code = NotFound desc = could not find container \"7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9\": container with ID starting with 7e7916786c3cfac95183236b8fa05442657a4a8e9b8112fe1b6fa5a6ea19a2d9 not found: ID does not exist"
Sep 07 10:20:29.265437 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.265415    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bc6pj"]
Sep 07 10:20:29.268689 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.268672    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bc6pj"]
Sep 07 10:20:29.894854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:29.894827    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0db54ffa-9853-4473-b568-63227b6a0622" path="/var/lib/kubelet/pods/0db54ffa-9853-4473-b568-63227b6a0622/volumes"
Sep 07 10:20:31.204487 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:31.204459    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:33.336363 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:33.336306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:33.435035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:33.435006    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:33.534198 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:33.534173    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:33.595082 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:33.594986    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:33.656937 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:33.656911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:36.909301 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:36.909266    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:20:37.010134 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:20:37.010104    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:10.245314 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:10.245261    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:10.390364 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:10.390314    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:11.626949 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:11.626909    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:11.729183 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:11.729137    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:11.828758 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:11.828723    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:11.928981 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:11.928913    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:12.278411 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:12.278340    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:12.378519 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:12.378491    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:12.494209 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:12.494178    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:21:21.683382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:21.683342    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:21.783535 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:21.783509    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:31.336435 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:31.336403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:31.438244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:31.438209    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:43.501241 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:43.501200    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:21:43.944780 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:21:43.944740    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:22:05.035933 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:05.035847    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:22:05.787992 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:05.787961    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:22:11.915992 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:11.915955    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:22:13.091443 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:13.091410    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:22:14.751161 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:14.751129    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:22:14.852012 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:14.851981    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:22:20.404251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.404222    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"]
Sep 07 10:22:20.404618 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.404526    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0db54ffa-9853-4473-b568-63227b6a0622" containerName="authorino"
Sep 07 10:22:20.404618 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.404537    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0db54ffa-9853-4473-b568-63227b6a0622" containerName="authorino"
Sep 07 10:22:20.404618 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.404599    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0db54ffa-9853-4473-b568-63227b6a0622" containerName="authorino"
Sep 07 10:22:20.407494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.407477    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.409825 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.409807    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 10:22:20.410773 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.410748    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 10:22:20.410867 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.410779    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-87q54\""
Sep 07 10:22:20.410867 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.410850    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 07 10:22:20.416192 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.416169    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"]
Sep 07 10:22:20.423530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.423501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:22:20.552981 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.552955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9f479b-fbcc-4127-bc1b-43467ee472db-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.553118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.552991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.553118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.553014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.553118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.553107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.553228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.553134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2gh\" (UniqueName: \"kubernetes.io/projected/fe9f479b-fbcc-4127-bc1b-43467ee472db-kube-api-access-nf2gh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.553228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.553158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654325 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654325 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2gh\" (UniqueName: \"kubernetes.io/projected/fe9f479b-fbcc-4127-bc1b-43467ee472db-kube-api-access-nf2gh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654325 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654603 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9f479b-fbcc-4127-bc1b-43467ee472db-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654603 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654603 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654449    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654764 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654764 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.654848 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.654781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.656517 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.656487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe9f479b-fbcc-4127-bc1b-43467ee472db-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.656754 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.656739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe9f479b-fbcc-4127-bc1b-43467ee472db-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.661773 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.661752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2gh\" (UniqueName: \"kubernetes.io/projected/fe9f479b-fbcc-4127-bc1b-43467ee472db-kube-api-access-nf2gh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szvgw\" (UID: \"fe9f479b-fbcc-4127-bc1b-43467ee472db\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.718399 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.718381    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:20.991600 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.989513    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"]
Sep 07 10:22:20.997407 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:22:20.997366    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9f479b_fbcc_4127_bc1b_43467ee472db.slice/crio-f07bada7dd00cb3b5a6167ed5b7a8ddd570540f2522d7f14ab079b2f3393a8ea WatchSource:0}: Error finding container f07bada7dd00cb3b5a6167ed5b7a8ddd570540f2522d7f14ab079b2f3393a8ea: Status 404 returned error can't find the container with id f07bada7dd00cb3b5a6167ed5b7a8ddd570540f2522d7f14ab079b2f3393a8ea
Sep 07 10:22:20.998399 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:20.998379    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:22:21.591968 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:21.591935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:22:21.719310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:21.719266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw" event={"ID":"fe9f479b-fbcc-4127-bc1b-43467ee472db","Type":"ContainerStarted","Data":"f07bada7dd00cb3b5a6167ed5b7a8ddd570540f2522d7f14ab079b2f3393a8ea"}
Sep 07 10:22:22.725541 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:22.725510    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw" event={"ID":"fe9f479b-fbcc-4127-bc1b-43467ee472db","Type":"ContainerStarted","Data":"0d3926c1ee268f48f908fa63b48793439dc25b78f758db6341362cd08a4b9358"}
Sep 07 10:22:22.742572 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:22.742521    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw" podStartSLOduration=1.694554411 podStartE2EDuration="2.742505884s" totalImagesPullingTime="1.047951473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:22:22.74225135 +0000 UTC m=+649.290766336" watchObservedRunningTime="2026-09-07 10:22:22.742505884 +0000 UTC m=+649.291020873"
Sep 07 10:22:30.718511 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:30.718463    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:30.718928 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:30.718542    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:30.730984 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:30.730960    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:30.768588 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:30.768559    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szvgw"
Sep 07 10:22:40.485844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:40.485807    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:22:41.994554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:41.994522    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:22:47.496455 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.496418    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"]
Sep 07 10:22:47.500387 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.500367    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.502713 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.502691    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 07 10:22:47.507917 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.507896    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"]
Sep 07 10:22:47.526120 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.526093    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:22:47.573359 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.573322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb1aecf-5642-4506-8280-3e97e235ea48-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.573551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.573368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.573551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.573392    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.573551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.573434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.573551 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.573506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.573727 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.573571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9m6t\" (UniqueName: \"kubernetes.io/projected/fbb1aecf-5642-4506-8280-3e97e235ea48-kube-api-access-x9m6t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674318 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9m6t\" (UniqueName: \"kubernetes.io/projected/fbb1aecf-5642-4506-8280-3e97e235ea48-kube-api-access-x9m6t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674318 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb1aecf-5642-4506-8280-3e97e235ea48-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674318 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674318 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674537 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674537 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674520    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674651 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674551    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.674651 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.674622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.676470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.676447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbb1aecf-5642-4506-8280-3e97e235ea48-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.676739 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.676723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb1aecf-5642-4506-8280-3e97e235ea48-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.681719 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.681695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9m6t\" (UniqueName: \"kubernetes.io/projected/fbb1aecf-5642-4506-8280-3e97e235ea48-kube-api-access-x9m6t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbstv\" (UID: \"fbb1aecf-5642-4506-8280-3e97e235ea48\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:47.811483 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:47.811455    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:48.080780 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:48.080757    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"]
Sep 07 10:22:48.083089 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:22:48.083042    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb1aecf_5642_4506_8280_3e97e235ea48.slice/crio-7004d4487fde1558c339a2a5af86a49f62809c25594a74f08a0c4f9996911f25 WatchSource:0}: Error finding container 7004d4487fde1558c339a2a5af86a49f62809c25594a74f08a0c4f9996911f25: Status 404 returned error can't find the container with id 7004d4487fde1558c339a2a5af86a49f62809c25594a74f08a0c4f9996911f25
Sep 07 10:22:48.836354 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:48.836322    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv" event={"ID":"fbb1aecf-5642-4506-8280-3e97e235ea48","Type":"ContainerStarted","Data":"463f540011cee346f2c77653a20ec6d45d1df045d1d1ce0b9ddc8732b7485423"}
Sep 07 10:22:48.836729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:48.836376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv" event={"ID":"fbb1aecf-5642-4506-8280-3e97e235ea48","Type":"ContainerStarted","Data":"7004d4487fde1558c339a2a5af86a49f62809c25594a74f08a0c4f9996911f25"}
Sep 07 10:22:48.853245 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:48.853201    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv" podStartSLOduration=1.651199093 podStartE2EDuration="1.85318564s" totalImagesPullingTime="201.986547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:22:48.852246207 +0000 UTC m=+675.400761195" watchObservedRunningTime="2026-09-07 10:22:48.85318564 +0000 UTC m=+675.401700628"
Sep 07 10:22:49.299991 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:49.299961    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:22:57.812561 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:57.812520    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:57.812984 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:57.812576    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:57.824793 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:57.824770    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:22:57.886791 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:22:57.886763    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbstv"
Sep 07 10:23:10.090497 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:10.090461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:23:12.497131 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:12.497095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:23:51.836508 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:51.836425    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:23:54.199863 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.199827    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79bc45b668-llbcs"]
Sep 07 10:23:54.203474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.203452    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79bc45b668-llbcs"
Sep 07 10:23:54.214136 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.214106    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79bc45b668-llbcs"]
Sep 07 10:23:54.319230 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.319207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87887868-71be-4133-a616-8a0c3c55010d-tls-cert\") pod \"authorino-79bc45b668-llbcs\" (UID: \"87887868-71be-4133-a616-8a0c3c55010d\") " pod="kuadrant-system/authorino-79bc45b668-llbcs"
Sep 07 10:23:54.319365 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.319247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr56z\" (UniqueName: \"kubernetes.io/projected/87887868-71be-4133-a616-8a0c3c55010d-kube-api-access-pr56z\") pod \"authorino-79bc45b668-llbcs\" (UID: \"87887868-71be-4133-a616-8a0c3c55010d\") " pod="kuadrant-system/authorino-79bc45b668-llbcs"
Sep 07 10:23:54.420408 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.420378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87887868-71be-4133-a616-8a0c3c55010d-tls-cert\") pod \"authorino-79bc45b668-llbcs\" (UID: \"87887868-71be-4133-a616-8a0c3c55010d\") " pod="kuadrant-system/authorino-79bc45b668-llbcs"
Sep 07 10:23:54.420520 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.420421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr56z\" (UniqueName: \"kubernetes.io/projected/87887868-71be-4133-a616-8a0c3c55010d-kube-api-access-pr56z\") pod \"authorino-79bc45b668-llbcs\" (UID: \"87887868-71be-4133-a616-8a0c3c55010d\") " pod="kuadrant-system/authorino-79bc45b668-llbcs"
Sep 07 10:23:54.422781 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.422764    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/87887868-71be-4133-a616-8a0c3c55010d-tls-cert\") pod \"authorino-79bc45b668-llbcs\" (UID: \"87887868-71be-4133-a616-8a0c3c55010d\") " pod="kuadrant-system/authorino-79bc45b668-llbcs"
Sep 07 10:23:54.428373 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.428344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr56z\" (UniqueName: \"kubernetes.io/projected/87887868-71be-4133-a616-8a0c3c55010d-kube-api-access-pr56z\") pod \"authorino-79bc45b668-llbcs\" (UID: \"87887868-71be-4133-a616-8a0c3c55010d\") " pod="kuadrant-system/authorino-79bc45b668-llbcs"
Sep 07 10:23:54.511952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.511900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79bc45b668-llbcs"
Sep 07 10:23:54.785433 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:54.785408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79bc45b668-llbcs"]
Sep 07 10:23:54.787250 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:23:54.787220    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87887868_71be_4133_a616_8a0c3c55010d.slice/crio-3db315c2e590f57c5eb0ef3c1266d375851892f60cc13eeec21ec974058e3ff8 WatchSource:0}: Error finding container 3db315c2e590f57c5eb0ef3c1266d375851892f60cc13eeec21ec974058e3ff8: Status 404 returned error can't find the container with id 3db315c2e590f57c5eb0ef3c1266d375851892f60cc13eeec21ec974058e3ff8
Sep 07 10:23:55.121373 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:55.121300    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79bc45b668-llbcs" event={"ID":"87887868-71be-4133-a616-8a0c3c55010d","Type":"ContainerStarted","Data":"3db315c2e590f57c5eb0ef3c1266d375851892f60cc13eeec21ec974058e3ff8"}
Sep 07 10:23:56.127279 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.127244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79bc45b668-llbcs" event={"ID":"87887868-71be-4133-a616-8a0c3c55010d","Type":"ContainerStarted","Data":"4bedff9447fb6d7d340cb6cd95832fe9734d931c69fb8752664dddf01368a1aa"}
Sep 07 10:23:56.155303 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.155239    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79bc45b668-llbcs" podStartSLOduration=1.552717983 podStartE2EDuration="2.155220876s" totalImagesPullingTime="602.502893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:23:56.154139422 +0000 UTC m=+742.702654411" watchObservedRunningTime="2026-09-07 10:23:56.155220876 +0000 UTC m=+742.703735862"
Sep 07 10:23:56.190923 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.190884    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78b89f7fd9-tqsrl"]
Sep 07 10:23:56.191153 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.191121    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl" podUID="934a6d8d-2653-4e4c-8ffa-d5641995eec6" containerName="authorino" containerID="cri-o://15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca" gracePeriod=30
Sep 07 10:23:56.454391 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.454365    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:23:56.537845 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.537802    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xj46\" (UniqueName: \"kubernetes.io/projected/934a6d8d-2653-4e4c-8ffa-d5641995eec6-kube-api-access-6xj46\") pod \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\" (UID: \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\") "
Sep 07 10:23:56.538016 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.537917    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/934a6d8d-2653-4e4c-8ffa-d5641995eec6-tls-cert\") pod \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\" (UID: \"934a6d8d-2653-4e4c-8ffa-d5641995eec6\") "
Sep 07 10:23:56.539806 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.539765    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934a6d8d-2653-4e4c-8ffa-d5641995eec6-kube-api-access-6xj46" (OuterVolumeSpecName: "kube-api-access-6xj46") pod "934a6d8d-2653-4e4c-8ffa-d5641995eec6" (UID: "934a6d8d-2653-4e4c-8ffa-d5641995eec6"). InnerVolumeSpecName "kube-api-access-6xj46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:23:56.547984 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.547959    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/934a6d8d-2653-4e4c-8ffa-d5641995eec6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "934a6d8d-2653-4e4c-8ffa-d5641995eec6" (UID: "934a6d8d-2653-4e4c-8ffa-d5641995eec6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:23:56.638835 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.638799    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/934a6d8d-2653-4e4c-8ffa-d5641995eec6-tls-cert\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:23:56.638835 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.638830    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xj46\" (UniqueName: \"kubernetes.io/projected/934a6d8d-2653-4e4c-8ffa-d5641995eec6-kube-api-access-6xj46\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:23:56.757021 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.756931    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:23:56.901756 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:56.901719    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:23:57.133420 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.133385    2560 generic.go:356] "Generic (PLEG): container finished" podID="934a6d8d-2653-4e4c-8ffa-d5641995eec6" containerID="15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca" exitCode=0
Sep 07 10:23:57.133852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.133432    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl"
Sep 07 10:23:57.133852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.133471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl" event={"ID":"934a6d8d-2653-4e4c-8ffa-d5641995eec6","Type":"ContainerDied","Data":"15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca"}
Sep 07 10:23:57.133852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.133516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b89f7fd9-tqsrl" event={"ID":"934a6d8d-2653-4e4c-8ffa-d5641995eec6","Type":"ContainerDied","Data":"9c9751e66e4591d067a5610f45b449467299741145021ce077690316c87c53dd"}
Sep 07 10:23:57.133852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.133536    2560 scope.go:117] "RemoveContainer" containerID="15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca"
Sep 07 10:23:57.142661 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.142637    2560 scope.go:117] "RemoveContainer" containerID="15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca"
Sep 07 10:23:57.142922 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:23:57.142896    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca\": container with ID starting with 15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca not found: ID does not exist" containerID="15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca"
Sep 07 10:23:57.143000 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.142932    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca"} err="failed to get container status \"15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca\": rpc error: code = NotFound desc = could not find container \"15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca\": container with ID starting with 15d32f95b777366b0e267296e30218ae2bd7661c39d950dc66a8e9bf767845ca not found: ID does not exist"
Sep 07 10:23:57.168024 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.167997    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78b89f7fd9-tqsrl"]
Sep 07 10:23:57.173585 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.173557    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78b89f7fd9-tqsrl"]
Sep 07 10:23:57.894679 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.894641    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934a6d8d-2653-4e4c-8ffa-d5641995eec6" path="/var/lib/kubelet/pods/934a6d8d-2653-4e4c-8ffa-d5641995eec6/volumes"
Sep 07 10:23:57.928503 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:57.928468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:23:58.068829 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:23:58.068789    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:07.681952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:07.681899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:07.745562 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:07.745522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:08.403937 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:08.403899    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:15.441627 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:15.441591    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:15.561166 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:15.561125    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:15.751536 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:15.751457    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:15.849363 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:15.849332    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:16.807686 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:16.807650    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:16.908677 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:16.908640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:17.269006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.268914    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:17.285856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.285828    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8026-139763500338352"]
Sep 07 10:24:17.286211 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.286198    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934a6d8d-2653-4e4c-8ffa-d5641995eec6" containerName="authorino"
Sep 07 10:24:17.286259 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.286213    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="934a6d8d-2653-4e4c-8ffa-d5641995eec6" containerName="authorino"
Sep 07 10:24:17.286294 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.286272    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="934a6d8d-2653-4e4c-8ffa-d5641995eec6" containerName="authorino"
Sep 07 10:24:17.289373 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.289358    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8026-139763500338352"
Sep 07 10:24:17.292000 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.291982    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qpjjb\""
Sep 07 10:24:17.306060 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.306038    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8026-139763500338352"]
Sep 07 10:24:17.421046 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.421006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqt5v\" (UniqueName: \"kubernetes.io/projected/6fa6bcee-4860-4921-a45c-1b0fc121c72d-kube-api-access-vqt5v\") pod \"test-curl-8026-139763500338352\" (UID: \"6fa6bcee-4860-4921-a45c-1b0fc121c72d\") " pod="opendatahub/test-curl-8026-139763500338352"
Sep 07 10:24:17.522549 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.522463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqt5v\" (UniqueName: \"kubernetes.io/projected/6fa6bcee-4860-4921-a45c-1b0fc121c72d-kube-api-access-vqt5v\") pod \"test-curl-8026-139763500338352\" (UID: \"6fa6bcee-4860-4921-a45c-1b0fc121c72d\") " pod="opendatahub/test-curl-8026-139763500338352"
Sep 07 10:24:17.531285 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.531261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqt5v\" (UniqueName: \"kubernetes.io/projected/6fa6bcee-4860-4921-a45c-1b0fc121c72d-kube-api-access-vqt5v\") pod \"test-curl-8026-139763500338352\" (UID: \"6fa6bcee-4860-4921-a45c-1b0fc121c72d\") " pod="opendatahub/test-curl-8026-139763500338352"
Sep 07 10:24:17.598904 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.598872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8026-139763500338352"
Sep 07 10:24:17.864902 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:17.864876    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8026-139763500338352"]
Sep 07 10:24:17.867362 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:24:17.867334    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fa6bcee_4860_4921_a45c_1b0fc121c72d.slice/crio-991bc5fa1845b92f42b5c0eb9cf9df145d80da2092f0eeabab38e3333abb6b3a WatchSource:0}: Error finding container 991bc5fa1845b92f42b5c0eb9cf9df145d80da2092f0eeabab38e3333abb6b3a: Status 404 returned error can't find the container with id 991bc5fa1845b92f42b5c0eb9cf9df145d80da2092f0eeabab38e3333abb6b3a
Sep 07 10:24:18.225379 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:18.225294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8026-139763500338352" event={"ID":"6fa6bcee-4860-4921-a45c-1b0fc121c72d","Type":"ContainerStarted","Data":"991bc5fa1845b92f42b5c0eb9cf9df145d80da2092f0eeabab38e3333abb6b3a"}
Sep 07 10:24:18.490355 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:18.490256    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:18.589462 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:18.589396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:18.985582 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:18.985498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:24:19.752153 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:19.752119    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:19.893835 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:19.893810    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:20.236880 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:20.236846    2560 generic.go:356] "Generic (PLEG): container finished" podID="6fa6bcee-4860-4921-a45c-1b0fc121c72d" containerID="73ead13029e070231b5dc6e87874b99652eb2fcdfc4a6dfa6072fd065590ffee" exitCode=0
Sep 07 10:24:20.237225 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:20.236929    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8026-139763500338352" event={"ID":"6fa6bcee-4860-4921-a45c-1b0fc121c72d","Type":"ContainerDied","Data":"73ead13029e070231b5dc6e87874b99652eb2fcdfc4a6dfa6072fd065590ffee"}
Sep 07 10:24:20.411333 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:20.411297    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:20.509421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:20.509328    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:21.362131 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.362109    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8026-139763500338352"
Sep 07 10:24:21.380969 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.380947    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-8026-139763500338352_6fa6bcee-4860-4921-a45c-1b0fc121c72d/test-curl-8026-139763500338352/0.log"
Sep 07 10:24:21.403339 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.403314    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8026-139763500338352"]
Sep 07 10:24:21.408341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.408314    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8026-139763500338352"]
Sep 07 10:24:21.458018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.457996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqt5v\" (UniqueName: \"kubernetes.io/projected/6fa6bcee-4860-4921-a45c-1b0fc121c72d-kube-api-access-vqt5v\") pod \"6fa6bcee-4860-4921-a45c-1b0fc121c72d\" (UID: \"6fa6bcee-4860-4921-a45c-1b0fc121c72d\") "
Sep 07 10:24:21.460092 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.460040    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa6bcee-4860-4921-a45c-1b0fc121c72d-kube-api-access-vqt5v" (OuterVolumeSpecName: "kube-api-access-vqt5v") pod "6fa6bcee-4860-4921-a45c-1b0fc121c72d" (UID: "6fa6bcee-4860-4921-a45c-1b0fc121c72d"). InnerVolumeSpecName "kube-api-access-vqt5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:24:21.521103 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.521032    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8026-139763500338096"]
Sep 07 10:24:21.521377 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.521363    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fa6bcee-4860-4921-a45c-1b0fc121c72d" containerName="test-curl-8026-139763500338352"
Sep 07 10:24:21.521416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.521379    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fa6bcee-4860-4921-a45c-1b0fc121c72d" containerName="test-curl-8026-139763500338352"
Sep 07 10:24:21.521450 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.521436    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fa6bcee-4860-4921-a45c-1b0fc121c72d" containerName="test-curl-8026-139763500338352"
Sep 07 10:24:21.524245 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.524231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8026-139763500338096"
Sep 07 10:24:21.530628 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.530608    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8026-139763500338096"]
Sep 07 10:24:21.559290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.559263    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqt5v\" (UniqueName: \"kubernetes.io/projected/6fa6bcee-4860-4921-a45c-1b0fc121c72d-kube-api-access-vqt5v\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:24:21.660014 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.659992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsmkw\" (UniqueName: \"kubernetes.io/projected/9517e642-5e7e-4fec-9924-80a54a295e6e-kube-api-access-jsmkw\") pod \"test-curl-8026-139763500338096\" (UID: \"9517e642-5e7e-4fec-9924-80a54a295e6e\") " pod="opendatahub/test-curl-8026-139763500338096"
Sep 07 10:24:21.761015 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.760994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsmkw\" (UniqueName: \"kubernetes.io/projected/9517e642-5e7e-4fec-9924-80a54a295e6e-kube-api-access-jsmkw\") pod \"test-curl-8026-139763500338096\" (UID: \"9517e642-5e7e-4fec-9924-80a54a295e6e\") " pod="opendatahub/test-curl-8026-139763500338096"
Sep 07 10:24:21.768297 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.768276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsmkw\" (UniqueName: \"kubernetes.io/projected/9517e642-5e7e-4fec-9924-80a54a295e6e-kube-api-access-jsmkw\") pod \"test-curl-8026-139763500338096\" (UID: \"9517e642-5e7e-4fec-9924-80a54a295e6e\") " pod="opendatahub/test-curl-8026-139763500338096"
Sep 07 10:24:21.832803 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.832781    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8026-139763500338096"
Sep 07 10:24:21.895976 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:21.895949    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fa6bcee-4860-4921-a45c-1b0fc121c72d" path="/var/lib/kubelet/pods/6fa6bcee-4860-4921-a45c-1b0fc121c72d/volumes"
Sep 07 10:24:22.099946 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:22.099883    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8026-139763500338096"]
Sep 07 10:24:22.102284 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:24:22.102256    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9517e642_5e7e_4fec_9924_80a54a295e6e.slice/crio-4fafbb938982744287e87f43b8b2de38bef34947b7fae65288327314a7c9ac11 WatchSource:0}: Error finding container 4fafbb938982744287e87f43b8b2de38bef34947b7fae65288327314a7c9ac11: Status 404 returned error can't find the container with id 4fafbb938982744287e87f43b8b2de38bef34947b7fae65288327314a7c9ac11
Sep 07 10:24:22.246528 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:22.246497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8026-139763500338096" event={"ID":"9517e642-5e7e-4fec-9924-80a54a295e6e","Type":"ContainerStarted","Data":"4fafbb938982744287e87f43b8b2de38bef34947b7fae65288327314a7c9ac11"}
Sep 07 10:24:22.247592 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:22.247571    2560 scope.go:117] "RemoveContainer" containerID="73ead13029e070231b5dc6e87874b99652eb2fcdfc4a6dfa6072fd065590ffee"
Sep 07 10:24:22.247735 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:22.247572    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8026-139763500338352"
Sep 07 10:24:22.952449 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:22.952421    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:23.014260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:23.014236    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:23.160709 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:23.160641    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:23.253715 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:23.253680    2560 generic.go:356] "Generic (PLEG): container finished" podID="9517e642-5e7e-4fec-9924-80a54a295e6e" containerID="9d9909fc8bc1cebf2a1551682411a472104b01a291c2b2fa352676470ae91962" exitCode=0
Sep 07 10:24:23.253845 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:23.253774    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8026-139763500338096" event={"ID":"9517e642-5e7e-4fec-9924-80a54a295e6e","Type":"ContainerDied","Data":"9d9909fc8bc1cebf2a1551682411a472104b01a291c2b2fa352676470ae91962"}
Sep 07 10:24:23.460982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:23.460905    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:23.576525 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:23.576492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:24.389820 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:24.389799    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8026-139763500338096"
Sep 07 10:24:24.408971 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:24.408948    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-8026-139763500338096_9517e642-5e7e-4fec-9924-80a54a295e6e/test-curl-8026-139763500338096/0.log"
Sep 07 10:24:24.427791 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:24.427741    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8026-139763500338096"]
Sep 07 10:24:24.430084 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:24.430053    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8026-139763500338096"]
Sep 07 10:24:24.480450 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:24.480427    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsmkw\" (UniqueName: \"kubernetes.io/projected/9517e642-5e7e-4fec-9924-80a54a295e6e-kube-api-access-jsmkw\") pod \"9517e642-5e7e-4fec-9924-80a54a295e6e\" (UID: \"9517e642-5e7e-4fec-9924-80a54a295e6e\") "
Sep 07 10:24:24.482382 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:24.482354    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9517e642-5e7e-4fec-9924-80a54a295e6e-kube-api-access-jsmkw" (OuterVolumeSpecName: "kube-api-access-jsmkw") pod "9517e642-5e7e-4fec-9924-80a54a295e6e" (UID: "9517e642-5e7e-4fec-9924-80a54a295e6e"). InnerVolumeSpecName "kube-api-access-jsmkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:24:24.582054 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:24.582029    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsmkw\" (UniqueName: \"kubernetes.io/projected/9517e642-5e7e-4fec-9924-80a54a295e6e-kube-api-access-jsmkw\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:24:24.915128 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:24.915032    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:25.221523 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:25.221440    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:25.264947 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:25.264920    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fafbb938982744287e87f43b8b2de38bef34947b7fae65288327314a7c9ac11"
Sep 07 10:24:25.264947 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:25.264921    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8026-139763500338096"
Sep 07 10:24:25.274673 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:24:25.274644    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8026-139763500338096\" is forbidden: User \"system:node:ip-10-0-128-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-191.ec2.internal' and this object" podUID="9517e642-5e7e-4fec-9924-80a54a295e6e" pod="opendatahub/test-curl-8026-139763500338096"
Sep 07 10:24:25.426456 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:25.426416    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:25.847261 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:25.847224    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:25.895386 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:25.895359    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9517e642-5e7e-4fec-9924-80a54a295e6e" path="/var/lib/kubelet/pods/9517e642-5e7e-4fec-9924-80a54a295e6e/volumes"
Sep 07 10:24:25.946386 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:25.946355    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:27.025779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.025747    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:27.126160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.126121    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:27.227962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.227933    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:27.618919 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.618882    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:27.717213 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.717179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:27.731498 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.731469    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"]
Sep 07 10:24:27.731787 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.731774    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9517e642-5e7e-4fec-9924-80a54a295e6e" containerName="test-curl-8026-139763500338096"
Sep 07 10:24:27.731827 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.731788    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9517e642-5e7e-4fec-9924-80a54a295e6e" containerName="test-curl-8026-139763500338096"
Sep 07 10:24:27.731862 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.731851    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9517e642-5e7e-4fec-9924-80a54a295e6e" containerName="test-curl-8026-139763500338096"
Sep 07 10:24:27.734887 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.734867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:27.737251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.737229    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8ec6b6b5\""
Sep 07 10:24:27.737354 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.737337    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8ec6b6b5-dockercfg-slnzg\""
Sep 07 10:24:27.744803 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.744781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"]
Sep 07 10:24:27.908431 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.908345    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cf6066-5ea7-470c-b0d5-019df60c9ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999\" (UID: \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:27.908431 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:27.908416    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwt7\" (UniqueName: \"kubernetes.io/projected/93cf6066-5ea7-470c-b0d5-019df60c9ebd-kube-api-access-5hwt7\") pod \"payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999\" (UID: \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:28.009922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:28.009882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cf6066-5ea7-470c-b0d5-019df60c9ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999\" (UID: \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:28.010219 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:28.010107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwt7\" (UniqueName: \"kubernetes.io/projected/93cf6066-5ea7-470c-b0d5-019df60c9ebd-kube-api-access-5hwt7\") pod \"payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999\" (UID: \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:28.010706 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:28.010683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cf6066-5ea7-470c-b0d5-019df60c9ebd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999\" (UID: \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:28.019416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:28.019385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwt7\" (UniqueName: \"kubernetes.io/projected/93cf6066-5ea7-470c-b0d5-019df60c9ebd-kube-api-access-5hwt7\") pod \"payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999\" (UID: \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:28.045319 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:28.045293    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:28.316597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:28.316574    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"]
Sep 07 10:24:28.319058 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:24:28.319033    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cf6066_5ea7_470c_b0d5_019df60c9ebd.slice/crio-7c611c195ccb62e7e1108b695cdd493930704c6dff382f3495a6d38dc7f45b49 WatchSource:0}: Error finding container 7c611c195ccb62e7e1108b695cdd493930704c6dff382f3495a6d38dc7f45b49: Status 404 returned error can't find the container with id 7c611c195ccb62e7e1108b695cdd493930704c6dff382f3495a6d38dc7f45b49
Sep 07 10:24:28.491185 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:28.491147    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:28.588978 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:28.588897    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:29.288282 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.288239    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999" event={"ID":"93cf6066-5ea7-470c-b0d5-019df60c9ebd","Type":"ContainerStarted","Data":"7c611c195ccb62e7e1108b695cdd493930704c6dff382f3495a6d38dc7f45b49"}
Sep 07 10:24:29.623482 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.623405    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"]
Sep 07 10:24:29.626986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.626966    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.630816 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.630790    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 10:24:29.632096 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.631939    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-ff5470-serving-cert\""
Sep 07 10:24:29.632096 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.631959    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 10:24:29.632096 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.632025    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2ldcm\""
Sep 07 10:24:29.632096 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.632051    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 10:24:29.643536 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.643512    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"]
Sep 07 10:24:29.669564 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.669512    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:29.726819 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.726783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7562\" (UniqueName: \"kubernetes.io/projected/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-kube-api-access-v7562\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.726987 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.726891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-openshift-service-ca\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.726987 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.726953    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-metrics-certs\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.727134 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.726998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-maas-api-tls\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.794312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.794274    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:29.828296 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.828256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7562\" (UniqueName: \"kubernetes.io/projected/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-kube-api-access-v7562\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.828480 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.828350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-openshift-service-ca\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.828480 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.828397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-metrics-certs\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.828480 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.828442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-maas-api-tls\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.829114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.829061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-openshift-service-ca\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.831256 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.831232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-metrics-certs\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.831256 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.831249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-maas-api-tls\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.837477 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.837449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7562\" (UniqueName: \"kubernetes.io/projected/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-kube-api-access-v7562\") pod \"maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:29.895008 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.894902    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:29.939548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:29.939509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:30.155179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.155093    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:30.257564 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.257519    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:30.304152 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.304119    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"]
Sep 07 10:24:30.311209 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.311187    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:30.313667 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.313642    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ff5470\""
Sep 07 10:24:30.313901 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.313874    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ff5470-dockercfg-j2r76\""
Sep 07 10:24:30.317662 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.317639    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"]
Sep 07 10:24:30.433669 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.433574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq87j\" (UniqueName: \"kubernetes.io/projected/5cf0d337-3048-4480-8f28-09fca608aab0-kube-api-access-zq87j\") pod \"payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx\" (UID: \"5cf0d337-3048-4480-8f28-09fca608aab0\") " pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:30.433823 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.433771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf0d337-3048-4480-8f28-09fca608aab0-plugins-config-volume\") pod \"payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx\" (UID: \"5cf0d337-3048-4480-8f28-09fca608aab0\") " pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:30.535177 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.535134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf0d337-3048-4480-8f28-09fca608aab0-plugins-config-volume\") pod \"payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx\" (UID: \"5cf0d337-3048-4480-8f28-09fca608aab0\") " pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:30.535369 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.535291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq87j\" (UniqueName: \"kubernetes.io/projected/5cf0d337-3048-4480-8f28-09fca608aab0-kube-api-access-zq87j\") pod \"payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx\" (UID: \"5cf0d337-3048-4480-8f28-09fca608aab0\") " pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:30.536029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.536006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf0d337-3048-4480-8f28-09fca608aab0-plugins-config-volume\") pod \"payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx\" (UID: \"5cf0d337-3048-4480-8f28-09fca608aab0\") " pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:30.546023 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.545989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq87j\" (UniqueName: \"kubernetes.io/projected/5cf0d337-3048-4480-8f28-09fca608aab0-kube-api-access-zq87j\") pod \"payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx\" (UID: \"5cf0d337-3048-4480-8f28-09fca608aab0\") " pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:30.625504 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:30.625457    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:31.053814 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.053735    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"]
Sep 07 10:24:31.055795 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:24:31.055761    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf0d337_3048_4480_8f28_09fca608aab0.slice/crio-4b5e6821795bcdc6b802c133c80afe8df5a7bfd182618dab4867b2ef6ab63a57 WatchSource:0}: Error finding container 4b5e6821795bcdc6b802c133c80afe8df5a7bfd182618dab4867b2ef6ab63a57: Status 404 returned error can't find the container with id 4b5e6821795bcdc6b802c133c80afe8df5a7bfd182618dab4867b2ef6ab63a57
Sep 07 10:24:31.056150 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.056124    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"]
Sep 07 10:24:31.059526 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:24:31.059504    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ac8c84_920a_4f74_acd0_2f8658a1c7d6.slice/crio-933e576beabc58587dc1794295b68766b8bfcfdeff260a01b7ab6229f06b8e42 WatchSource:0}: Error finding container 933e576beabc58587dc1794295b68766b8bfcfdeff260a01b7ab6229f06b8e42: Status 404 returned error can't find the container with id 933e576beabc58587dc1794295b68766b8bfcfdeff260a01b7ab6229f06b8e42
Sep 07 10:24:31.301582 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.301551    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj" event={"ID":"44ac8c84-920a-4f74-acd0-2f8658a1c7d6","Type":"ContainerStarted","Data":"933e576beabc58587dc1794295b68766b8bfcfdeff260a01b7ab6229f06b8e42"}
Sep 07 10:24:31.303003 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.302932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx" event={"ID":"5cf0d337-3048-4480-8f28-09fca608aab0","Type":"ContainerStarted","Data":"4cd4c5d6a86c88b18f70aebd678171f649cad372a220b7bece51f1b4bc7885a1"}
Sep 07 10:24:31.303003 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.302978    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx" event={"ID":"5cf0d337-3048-4480-8f28-09fca608aab0","Type":"ContainerStarted","Data":"4b5e6821795bcdc6b802c133c80afe8df5a7bfd182618dab4867b2ef6ab63a57"}
Sep 07 10:24:31.303205 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.303009    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:31.304482 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.304463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999" event={"ID":"93cf6066-5ea7-470c-b0d5-019df60c9ebd","Type":"ContainerStarted","Data":"da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618"}
Sep 07 10:24:31.304779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.304518    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:31.320364 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.320312    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx" podStartSLOduration=1.3202986 podStartE2EDuration="1.3202986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:31.318758609 +0000 UTC m=+777.867273597" watchObservedRunningTime="2026-09-07 10:24:31.3202986 +0000 UTC m=+777.868813618"
Sep 07 10:24:31.334664 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:31.334616    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999" podStartSLOduration=1.811439394 podStartE2EDuration="4.334602733s" totalImagesPullingTime="2.523163339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:31.333435131 +0000 UTC m=+777.881950119" watchObservedRunningTime="2026-09-07 10:24:31.334602733 +0000 UTC m=+777.883117719"
Sep 07 10:24:33.197215 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:33.197177    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:24:33.316487 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:33.316452    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj" event={"ID":"44ac8c84-920a-4f74-acd0-2f8658a1c7d6","Type":"ContainerStarted","Data":"304a52ec7dd80a9c56e4851b1b9149c6080ec5a08318654acd4c124f39bc7de4"}
Sep 07 10:24:33.316632 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:33.316589    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:33.335088 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:33.335024    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj" podStartSLOduration=2.761421178 podStartE2EDuration="4.335004906s" totalImagesPullingTime="1.573583728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:33.333722797 +0000 UTC m=+779.882237785" watchObservedRunningTime="2026-09-07 10:24:33.335004906 +0000 UTC m=+779.883519895"
Sep 07 10:24:34.119059 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:34.118915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:34.259422 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:34.259385    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:34.352355 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:34.352324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:34.453092 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:34.452998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:37.150870 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:37.150832    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:38.617380 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:38.617343    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:39.325506 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.325479    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:24:39.582770 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.582700    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"]
Sep 07 10:24:39.587212 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.587195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.589503 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.589486    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-5bc2bc81-kserve-self-signed-certs\""
Sep 07 10:24:39.594490 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.594467    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"]
Sep 07 10:24:39.610146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.610124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kfm\" (UniqueName: \"kubernetes.io/projected/6d548bca-247f-4908-886e-14e4395acb66-kube-api-access-k7kfm\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.610264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.610154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-home\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.610264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.610172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-tmp-dir\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.610264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.610191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-dshm\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.610375 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.610285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-model-cache\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.610375 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.610343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d548bca-247f-4908-886e-14e4395acb66-tls-certs\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.711499 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.711464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kfm\" (UniqueName: \"kubernetes.io/projected/6d548bca-247f-4908-886e-14e4395acb66-kube-api-access-k7kfm\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.711499 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.711503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-home\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.711968 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.711521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-tmp-dir\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.711968 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.711562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-dshm\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.711968 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.711589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-model-cache\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.711968 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.711626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d548bca-247f-4908-886e-14e4395acb66-tls-certs\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.711968 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.711847    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-home\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.711968 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.711927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-tmp-dir\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.712239 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.712009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-model-cache\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.713923 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.713898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-dshm\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.714057 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.714014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d548bca-247f-4908-886e-14e4395acb66-tls-certs\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.720146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.720124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kfm\" (UniqueName: \"kubernetes.io/projected/6d548bca-247f-4908-886e-14e4395acb66-kube-api-access-k7kfm\") pod \"e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") " pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:39.884427 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.884352    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:39.896974 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:39.896940    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:40.166769 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:40.166741    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"]
Sep 07 10:24:40.169377 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:24:40.169337    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d548bca_247f_4908_886e_14e4395acb66.slice/crio-135e651d0ea68e705b289649067a2f9cfbaaeef39f97f8002545c4396adc42fd WatchSource:0}: Error finding container 135e651d0ea68e705b289649067a2f9cfbaaeef39f97f8002545c4396adc42fd: Status 404 returned error can't find the container with id 135e651d0ea68e705b289649067a2f9cfbaaeef39f97f8002545c4396adc42fd
Sep 07 10:24:40.348748 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:40.348716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k" event={"ID":"6d548bca-247f-4908-886e-14e4395acb66","Type":"ContainerStarted","Data":"3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878"}
Sep 07 10:24:40.348920 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:40.348758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k" event={"ID":"6d548bca-247f-4908-886e-14e4395acb66","Type":"ContainerStarted","Data":"135e651d0ea68e705b289649067a2f9cfbaaeef39f97f8002545c4396adc42fd"}
Sep 07 10:24:40.367793 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:40.367746    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k" podStartSLOduration=1.367731144 podStartE2EDuration="1.367731144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:40.36687093 +0000 UTC m=+786.915385917" watchObservedRunningTime="2026-09-07 10:24:40.367731144 +0000 UTC m=+786.916246130"
Sep 07 10:24:40.799310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:40.799269    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:42.311753 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:42.311724    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:24:42.312210 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:42.311793    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:24:43.304840 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:43.304805    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:43.615647 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:43.615536    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:43.986962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:43.986866    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:44.344457 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:44.344425    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:47.603967 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:47.603927    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:47.703414 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:47.703379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:48.156123 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.156086    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:48.903376 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.903340    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"]
Sep 07 10:24:48.907354 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.907334    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:48.909718 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.909697    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f17c1f-serving-cert\""
Sep 07 10:24:48.916705 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.916679    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"]
Sep 07 10:24:48.935791 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.935752    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:48.996617 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.996588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ck4\" (UniqueName: \"kubernetes.io/projected/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-kube-api-access-w4ck4\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:48.996748 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.996681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:48.996748 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.996707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-metrics-certs\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:48.996748 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:48.996726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.037825 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.037795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:49.097872 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.097845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ck4\" (UniqueName: \"kubernetes.io/projected/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-kube-api-access-w4ck4\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.098000 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.097907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.098000 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.097935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-metrics-certs\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.098000 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.097962    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.098193 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:24:49.098124    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-serving-cert: secret "maas-api-e2e-inf-b-f17c1f-serving-cert" not found
Sep 07 10:24:49.098193 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:24:49.098186    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls podName:8ec0645f-1c1a-4e1c-9d02-f954812e9a72 nodeName:}" failed. No retries permitted until 2026-09-07 10:24:49.598166701 +0000 UTC m=+796.146681680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls") pod "maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x" (UID: "8ec0645f-1c1a-4e1c-9d02-f954812e9a72") : secret "maas-api-e2e-inf-b-f17c1f-serving-cert" not found
Sep 07 10:24:49.098659 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.098635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.100407 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.100387    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-metrics-certs\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.106563 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.106534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ck4\" (UniqueName: \"kubernetes.io/projected/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-kube-api-access-w4ck4\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.335348 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.335312    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:49.511139 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.511110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:49.601731 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.601645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.603939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.603913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls\") pod \"maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.610609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.610578    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:49.639788 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.639760    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"]
Sep 07 10:24:49.649446 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.649427    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:49.652307 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.652280    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f17c1f\""
Sep 07 10:24:49.652307 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.652299    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f17c1f-dockercfg-xmck7\""
Sep 07 10:24:49.653885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.653862    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"]
Sep 07 10:24:49.702800 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.702768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07757aab-510d-478f-be6e-ad0a4f64f1a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b\" (UID: \"07757aab-510d-478f-be6e-ad0a4f64f1a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:49.702978 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.702812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5xjc\" (UniqueName: \"kubernetes.io/projected/07757aab-510d-478f-be6e-ad0a4f64f1a3-kube-api-access-w5xjc\") pod \"payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b\" (UID: \"07757aab-510d-478f-be6e-ad0a4f64f1a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:49.804312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.804270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07757aab-510d-478f-be6e-ad0a4f64f1a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b\" (UID: \"07757aab-510d-478f-be6e-ad0a4f64f1a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:49.804312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.804314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5xjc\" (UniqueName: \"kubernetes.io/projected/07757aab-510d-478f-be6e-ad0a4f64f1a3-kube-api-access-w5xjc\") pod \"payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b\" (UID: \"07757aab-510d-478f-be6e-ad0a4f64f1a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:49.804999 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.804978    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07757aab-510d-478f-be6e-ad0a4f64f1a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b\" (UID: \"07757aab-510d-478f-be6e-ad0a4f64f1a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:49.812457 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.812411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5xjc\" (UniqueName: \"kubernetes.io/projected/07757aab-510d-478f-be6e-ad0a4f64f1a3-kube-api-access-w5xjc\") pod \"payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b\" (UID: \"07757aab-510d-478f-be6e-ad0a4f64f1a3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:49.820308 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.820283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:49.898575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.897304    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:49.898575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.897592    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:49.910783 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.910755    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:49.927034 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.927002    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:49.961460 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:49.961426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:50.092910 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.092867    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"]
Sep 07 10:24:50.095505 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:24:50.095479    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec0645f_1c1a_4e1c_9d02_f954812e9a72.slice/crio-dc4ad62f924370a2ce8e3b2d4edae819811608bc7412fc8d525db91943b3091a WatchSource:0}: Error finding container dc4ad62f924370a2ce8e3b2d4edae819811608bc7412fc8d525db91943b3091a: Status 404 returned error can't find the container with id dc4ad62f924370a2ce8e3b2d4edae819811608bc7412fc8d525db91943b3091a
Sep 07 10:24:50.235265 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.235241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"]
Sep 07 10:24:50.237343 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:24:50.237316    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07757aab_510d_478f_be6e_ad0a4f64f1a3.slice/crio-f0bf1462138ab1f1c1b3f0c107b5478e1440d075b5395e809575574242256e51 WatchSource:0}: Error finding container f0bf1462138ab1f1c1b3f0c107b5478e1440d075b5395e809575574242256e51: Status 404 returned error can't find the container with id f0bf1462138ab1f1c1b3f0c107b5478e1440d075b5395e809575574242256e51
Sep 07 10:24:50.397751 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.397715    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x" event={"ID":"8ec0645f-1c1a-4e1c-9d02-f954812e9a72","Type":"ContainerStarted","Data":"dc4ad62f924370a2ce8e3b2d4edae819811608bc7412fc8d525db91943b3091a"}
Sep 07 10:24:50.399398 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.399370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b" event={"ID":"07757aab-510d-478f-be6e-ad0a4f64f1a3","Type":"ContainerStarted","Data":"c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2"}
Sep 07 10:24:50.399522 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.399419    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b" event={"ID":"07757aab-510d-478f-be6e-ad0a4f64f1a3","Type":"ContainerStarted","Data":"f0bf1462138ab1f1c1b3f0c107b5478e1440d075b5395e809575574242256e51"}
Sep 07 10:24:50.399592 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.399537    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:24:50.410645 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.410620    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:24:50.416506 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.416461    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b" podStartSLOduration=1.416448452 podStartE2EDuration="1.416448452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:50.415276003 +0000 UTC m=+796.963790991" watchObservedRunningTime="2026-09-07 10:24:50.416448452 +0000 UTC m=+796.964963438"
Sep 07 10:24:50.503139 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.503015    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:24:50.752636 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:50.752600    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:51.409375 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:51.409336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x" event={"ID":"8ec0645f-1c1a-4e1c-9d02-f954812e9a72","Type":"ContainerStarted","Data":"2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6"}
Sep 07 10:24:51.409846 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:51.409827    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:51.525519 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:51.525460    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:52.592986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:52.592947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:53.373871 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:53.373836    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:53.544041 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:53.544011    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:24:58.422228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:58.422152    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:24:58.440330 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:24:58.440274    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x" podStartSLOduration=9.979118483 podStartE2EDuration="10.440257137s" totalImagesPullingTime="461.138654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:24:51.43181476 +0000 UTC m=+797.980329748" watchObservedRunningTime="2026-09-07 10:24:58.440257137 +0000 UTC m=+804.988772123"
Sep 07 10:25:01.411630 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:01.411587    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:25:04.496705 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:04.496675    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:25:07.330142 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:07.330110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:07.791295 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:07.791263    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:07.898999 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:07.898965    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:08.010258 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.010229    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:08.047275 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.047199    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"]
Sep 07 10:25:08.047417 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.047388    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999" podUID="93cf6066-5ea7-470c-b0d5-019df60c9ebd" containerName="payload-pre-processing" containerID="cri-o://da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618" gracePeriod=30
Sep 07 10:25:08.100604 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.100553    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:08.200545 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.200508    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:08.296160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.296134    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:25:08.300278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.300194    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:08.365253 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.365221    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:08.473815 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.473775    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hwt7\" (UniqueName: \"kubernetes.io/projected/93cf6066-5ea7-470c-b0d5-019df60c9ebd-kube-api-access-5hwt7\") pod \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\" (UID: \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\") "
Sep 07 10:25:08.474042 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.473837    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cf6066-5ea7-470c-b0d5-019df60c9ebd-plugins-config-volume\") pod \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\" (UID: \"93cf6066-5ea7-470c-b0d5-019df60c9ebd\") "
Sep 07 10:25:08.474230 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.474208    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93cf6066-5ea7-470c-b0d5-019df60c9ebd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "93cf6066-5ea7-470c-b0d5-019df60c9ebd" (UID: "93cf6066-5ea7-470c-b0d5-019df60c9ebd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:25:08.476017 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.475996    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cf6066-5ea7-470c-b0d5-019df60c9ebd-kube-api-access-5hwt7" (OuterVolumeSpecName: "kube-api-access-5hwt7") pod "93cf6066-5ea7-470c-b0d5-019df60c9ebd" (UID: "93cf6066-5ea7-470c-b0d5-019df60c9ebd"). InnerVolumeSpecName "kube-api-access-5hwt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:25:08.488088 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.488038    2560 generic.go:356] "Generic (PLEG): container finished" podID="93cf6066-5ea7-470c-b0d5-019df60c9ebd" containerID="da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618" exitCode=0
Sep 07 10:25:08.488205 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.488091    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"
Sep 07 10:25:08.488205 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.488106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999" event={"ID":"93cf6066-5ea7-470c-b0d5-019df60c9ebd","Type":"ContainerDied","Data":"da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618"}
Sep 07 10:25:08.488205 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.488158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999" event={"ID":"93cf6066-5ea7-470c-b0d5-019df60c9ebd","Type":"ContainerDied","Data":"7c611c195ccb62e7e1108b695cdd493930704c6dff382f3495a6d38dc7f45b49"}
Sep 07 10:25:08.488205 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.488181    2560 scope.go:117] "RemoveContainer" containerID="da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618"
Sep 07 10:25:08.497179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.497155    2560 scope.go:117] "RemoveContainer" containerID="da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618"
Sep 07 10:25:08.497435 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:25:08.497420    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618\": container with ID starting with da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618 not found: ID does not exist" containerID="da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618"
Sep 07 10:25:08.497488 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.497440    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618"} err="failed to get container status \"da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618\": rpc error: code = NotFound desc = could not find container \"da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618\": container with ID starting with da5fa200ed05b9ee65bcc65d2937ddec9edd1612516a6467489391566656a618 not found: ID does not exist"
Sep 07 10:25:08.509439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.509414    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"]
Sep 07 10:25:08.511819 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.511801    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8ec6b6b5-6f995c7f58-mz999"]
Sep 07 10:25:08.575166 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.575049    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwt7\" (UniqueName: \"kubernetes.io/projected/93cf6066-5ea7-470c-b0d5-019df60c9ebd-kube-api-access-5hwt7\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:25:08.575166 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.575107    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93cf6066-5ea7-470c-b0d5-019df60c9ebd-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:25:08.790277 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.790237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:08.883671 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.883580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:08.975091 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:08.975043    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:09.269552 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:09.269465    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:09.562543 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:09.562510    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:09.895648 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:09.895574    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93cf6066-5ea7-470c-b0d5-019df60c9ebd" path="/var/lib/kubelet/pods/93cf6066-5ea7-470c-b0d5-019df60c9ebd/volumes"
Sep 07 10:25:09.899981 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:09.899951    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:12.318378 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:12.318335    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:12.413866 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:12.413833    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:12.492619 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:12.492592    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:12.745930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:12.745838    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:12.847687 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:12.847638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:13.132127 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.132051    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:13.225187 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.225157    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:13.269167 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.269137    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"]
Sep 07 10:25:13.269501 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.269490    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93cf6066-5ea7-470c-b0d5-019df60c9ebd" containerName="payload-pre-processing"
Sep 07 10:25:13.269543 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.269504    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="93cf6066-5ea7-470c-b0d5-019df60c9ebd" containerName="payload-pre-processing"
Sep 07 10:25:13.269582 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.269566    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="93cf6066-5ea7-470c-b0d5-019df60c9ebd" containerName="payload-pre-processing"
Sep 07 10:25:13.274466 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.274448    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:13.276950 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.276911    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-de6561a9\""
Sep 07 10:25:13.277080 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.276969    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-de6561a9-dockercfg-sn7cj\""
Sep 07 10:25:13.286379 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.286352    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"]
Sep 07 10:25:13.418315 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.418222    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87a9ee2-90e7-485e-9875-8ae42c41552d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz\" (UID: \"a87a9ee2-90e7-485e-9875-8ae42c41552d\") " pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:13.418315 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.418276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b89hz\" (UniqueName: \"kubernetes.io/projected/a87a9ee2-90e7-485e-9875-8ae42c41552d-kube-api-access-b89hz\") pod \"payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz\" (UID: \"a87a9ee2-90e7-485e-9875-8ae42c41552d\") " pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:13.518754 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.518722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87a9ee2-90e7-485e-9875-8ae42c41552d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz\" (UID: \"a87a9ee2-90e7-485e-9875-8ae42c41552d\") " pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:13.518894 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.518758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b89hz\" (UniqueName: \"kubernetes.io/projected/a87a9ee2-90e7-485e-9875-8ae42c41552d-kube-api-access-b89hz\") pod \"payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz\" (UID: \"a87a9ee2-90e7-485e-9875-8ae42c41552d\") " pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:13.519632 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.519608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87a9ee2-90e7-485e-9875-8ae42c41552d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz\" (UID: \"a87a9ee2-90e7-485e-9875-8ae42c41552d\") " pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:13.532581 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.532557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b89hz\" (UniqueName: \"kubernetes.io/projected/a87a9ee2-90e7-485e-9875-8ae42c41552d-kube-api-access-b89hz\") pod \"payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz\" (UID: \"a87a9ee2-90e7-485e-9875-8ae42c41552d\") " pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:13.584926 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.584901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:13.851884 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:13.851827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"]
Sep 07 10:25:13.853965 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:25:13.853938    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87a9ee2_90e7_485e_9875_8ae42c41552d.slice/crio-edf26c86712823f10e6985b64db609e65c86200c2fbe12c9e2a579ee8454fcd3 WatchSource:0}: Error finding container edf26c86712823f10e6985b64db609e65c86200c2fbe12c9e2a579ee8454fcd3: Status 404 returned error can't find the container with id edf26c86712823f10e6985b64db609e65c86200c2fbe12c9e2a579ee8454fcd3
Sep 07 10:25:14.517675 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:14.517639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz" event={"ID":"a87a9ee2-90e7-485e-9875-8ae42c41552d","Type":"ContainerStarted","Data":"6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637"}
Sep 07 10:25:14.518035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:14.517698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz" event={"ID":"a87a9ee2-90e7-485e-9875-8ae42c41552d","Type":"ContainerStarted","Data":"edf26c86712823f10e6985b64db609e65c86200c2fbe12c9e2a579ee8454fcd3"}
Sep 07 10:25:14.518035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:14.517733    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:14.533507 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:14.533451    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz" podStartSLOduration=1.5334364059999999 podStartE2EDuration="1.533436406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:14.532617664 +0000 UTC m=+821.081132664" watchObservedRunningTime="2026-09-07 10:25:14.533436406 +0000 UTC m=+821.081951393"
Sep 07 10:25:21.096444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:21.096403    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:25:25.524252 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:25.524215    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:26.304502 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:26.304462    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:26.405108 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:26.405046    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:26.518548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:26.518522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:32.360980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:32.360942    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:32.424174 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:32.424142    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:32.498431 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:32.498406    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:32.622499 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:32.622433    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:32.728956 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:32.728924    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:35.491405 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:35.491365    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:38.458436 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:38.458403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:38.557016 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:38.556985    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:39.391237 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:39.391201    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:25:43.304431 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:43.304386    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:43.953389 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:43.953355    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:43.982937 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:43.982910    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"]
Sep 07 10:25:43.983209 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:43.983182    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz" podUID="a87a9ee2-90e7-485e-9875-8ae42c41552d" containerName="payload-processing" containerID="cri-o://6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637" gracePeriod=30
Sep 07 10:25:44.054252 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.054220    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:44.154867 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.154834    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:44.234636 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.234613    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:44.255251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.255206    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:44.323712 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.323669    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:44.356159 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.356135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87a9ee2-90e7-485e-9875-8ae42c41552d-plugins-config-volume\") pod \"a87a9ee2-90e7-485e-9875-8ae42c41552d\" (UID: \"a87a9ee2-90e7-485e-9875-8ae42c41552d\") "
Sep 07 10:25:44.356269 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.356165    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b89hz\" (UniqueName: \"kubernetes.io/projected/a87a9ee2-90e7-485e-9875-8ae42c41552d-kube-api-access-b89hz\") pod \"a87a9ee2-90e7-485e-9875-8ae42c41552d\" (UID: \"a87a9ee2-90e7-485e-9875-8ae42c41552d\") "
Sep 07 10:25:44.356513 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.356492    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87a9ee2-90e7-485e-9875-8ae42c41552d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a87a9ee2-90e7-485e-9875-8ae42c41552d" (UID: "a87a9ee2-90e7-485e-9875-8ae42c41552d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:25:44.358212 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.358191    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87a9ee2-90e7-485e-9875-8ae42c41552d-kube-api-access-b89hz" (OuterVolumeSpecName: "kube-api-access-b89hz") pod "a87a9ee2-90e7-485e-9875-8ae42c41552d" (UID: "a87a9ee2-90e7-485e-9875-8ae42c41552d"). InnerVolumeSpecName "kube-api-access-b89hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:25:44.457046 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.457016    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87a9ee2-90e7-485e-9875-8ae42c41552d-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:25:44.457046 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.457041    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b89hz\" (UniqueName: \"kubernetes.io/projected/a87a9ee2-90e7-485e-9875-8ae42c41552d-kube-api-access-b89hz\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:25:44.655476 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.655444    2560 generic.go:356] "Generic (PLEG): container finished" podID="a87a9ee2-90e7-485e-9875-8ae42c41552d" containerID="6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637" exitCode=0
Sep 07 10:25:44.655652 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.655480    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"
Sep 07 10:25:44.655652 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.655534    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz" event={"ID":"a87a9ee2-90e7-485e-9875-8ae42c41552d","Type":"ContainerDied","Data":"6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637"}
Sep 07 10:25:44.655652 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.655587    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz" event={"ID":"a87a9ee2-90e7-485e-9875-8ae42c41552d","Type":"ContainerDied","Data":"edf26c86712823f10e6985b64db609e65c86200c2fbe12c9e2a579ee8454fcd3"}
Sep 07 10:25:44.655652 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.655615    2560 scope.go:117] "RemoveContainer" containerID="6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637"
Sep 07 10:25:44.663956 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.663940    2560 scope.go:117] "RemoveContainer" containerID="6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637"
Sep 07 10:25:44.664262 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:25:44.664241    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637\": container with ID starting with 6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637 not found: ID does not exist" containerID="6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637"
Sep 07 10:25:44.664312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.664270    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637"} err="failed to get container status \"6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637\": rpc error: code = NotFound desc = could not find container \"6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637\": container with ID starting with 6b351a9b917fa0891e6d32c172910f84142965c3eb2db055043af7f46e22d637 not found: ID does not exist"
Sep 07 10:25:44.675673 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.675654    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"]
Sep 07 10:25:44.679216 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.679197    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-de6561a9-67d985585d-tjtvz"]
Sep 07 10:25:44.722946 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:44.722916    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:45.018807 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:45.018730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:45.118784 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:45.118752    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:45.895648 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:45.895618    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87a9ee2-90e7-485e-9875-8ae42c41552d" path="/var/lib/kubelet/pods/a87a9ee2-90e7-485e-9875-8ae42c41552d/volumes"
Sep 07 10:25:47.518922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:47.518882    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:47.614762 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:47.614731    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:48.830245 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:48.830205    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:48.930286 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:48.930251    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:49.032815 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.032788    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:49.296351 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.296316    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:49.395456 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.395420    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:25:49.434273 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.434242    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"]
Sep 07 10:25:49.434632 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.434620    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87a9ee2-90e7-485e-9875-8ae42c41552d" containerName="payload-processing"
Sep 07 10:25:49.434682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.434634    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87a9ee2-90e7-485e-9875-8ae42c41552d" containerName="payload-processing"
Sep 07 10:25:49.434730 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.434709    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87a9ee2-90e7-485e-9875-8ae42c41552d" containerName="payload-processing"
Sep 07 10:25:49.439182 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.439159    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:49.442031 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.442011    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-31186eae\""
Sep 07 10:25:49.442257 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.442243    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-31186eae-dockercfg-2rp8d\""
Sep 07 10:25:49.448430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.448408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"]
Sep 07 10:25:49.500997 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.500966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd066e6-eff6-4709-b668-d7e02bf79808-plugins-config-volume\") pod \"payload-processing-e2e-ait-31186eae-f5d9dd75-j795r\" (UID: \"7cd066e6-eff6-4709-b668-d7e02bf79808\") " pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:49.501180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.501013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddghw\" (UniqueName: \"kubernetes.io/projected/7cd066e6-eff6-4709-b668-d7e02bf79808-kube-api-access-ddghw\") pod \"payload-processing-e2e-ait-31186eae-f5d9dd75-j795r\" (UID: \"7cd066e6-eff6-4709-b668-d7e02bf79808\") " pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:49.602040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.601966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd066e6-eff6-4709-b668-d7e02bf79808-plugins-config-volume\") pod \"payload-processing-e2e-ait-31186eae-f5d9dd75-j795r\" (UID: \"7cd066e6-eff6-4709-b668-d7e02bf79808\") " pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:49.602040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.602003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddghw\" (UniqueName: \"kubernetes.io/projected/7cd066e6-eff6-4709-b668-d7e02bf79808-kube-api-access-ddghw\") pod \"payload-processing-e2e-ait-31186eae-f5d9dd75-j795r\" (UID: \"7cd066e6-eff6-4709-b668-d7e02bf79808\") " pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:49.602825 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.602804    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd066e6-eff6-4709-b668-d7e02bf79808-plugins-config-volume\") pod \"payload-processing-e2e-ait-31186eae-f5d9dd75-j795r\" (UID: \"7cd066e6-eff6-4709-b668-d7e02bf79808\") " pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:49.610051 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.610021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddghw\" (UniqueName: \"kubernetes.io/projected/7cd066e6-eff6-4709-b668-d7e02bf79808-kube-api-access-ddghw\") pod \"payload-processing-e2e-ait-31186eae-f5d9dd75-j795r\" (UID: \"7cd066e6-eff6-4709-b668-d7e02bf79808\") " pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:49.750444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:49.750417    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:50.021234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:50.021206    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"]
Sep 07 10:25:50.023580 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:25:50.023552    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd066e6_eff6_4709_b668_d7e02bf79808.slice/crio-65cd1e73382e34dc3140b70cc1a9f209c24a8df442bbeec3c59e733633fcf461 WatchSource:0}: Error finding container 65cd1e73382e34dc3140b70cc1a9f209c24a8df442bbeec3c59e733633fcf461: Status 404 returned error can't find the container with id 65cd1e73382e34dc3140b70cc1a9f209c24a8df442bbeec3c59e733633fcf461
Sep 07 10:25:50.684735 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:50.684703    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r" event={"ID":"7cd066e6-eff6-4709-b668-d7e02bf79808","Type":"ContainerStarted","Data":"63fe6018e6d6d2d6c97f602ca5134c12956f3679446debacc8d76a1d656d216b"}
Sep 07 10:25:50.684917 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:50.684771    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r" event={"ID":"7cd066e6-eff6-4709-b668-d7e02bf79808","Type":"ContainerStarted","Data":"65cd1e73382e34dc3140b70cc1a9f209c24a8df442bbeec3c59e733633fcf461"}
Sep 07 10:25:50.684917 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:50.684850    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:25:50.700414 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:25:50.700357    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r" podStartSLOduration=1.70034435 podStartE2EDuration="1.70034435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:25:50.699727545 +0000 UTC m=+857.248242533" watchObservedRunningTime="2026-09-07 10:25:50.70034435 +0000 UTC m=+857.248859338"
Sep 07 10:26:00.600117 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:00.600085    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:26:01.692663 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:01.692630    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:26:07.786400 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.786366    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"]
Sep 07 10:26:07.789948 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.789920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:07.792400 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.792381    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-05ecbf\"/\"openshift-service-ca.crt\""
Sep 07 10:26:07.793676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.793654    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-05ecbf\"/\"test-model-05ecbf-kserve-self-signed-certs\""
Sep 07 10:26:07.793788 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.793690    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-05ecbf\"/\"default-dockercfg-vdz5s\""
Sep 07 10:26:07.793788 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.793744    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-05ecbf\"/\"kube-root-ca.crt\""
Sep 07 10:26:07.797326 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.797308    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"]
Sep 07 10:26:07.812255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.812215    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:07.948756 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.948724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-model-cache\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:07.948756 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.948770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-home\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:07.949003 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.948793    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tmp-dir\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:07.949003 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.948814    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6tj\" (UniqueName: \"kubernetes.io/projected/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-kube-api-access-5l6tj\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:07.949003 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.948850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-dshm\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:07.949003 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:07.948960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tls-certs\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050434 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tls-certs\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-model-cache\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-home\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tmp-dir\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6tj\" (UniqueName: \"kubernetes.io/projected/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-kube-api-access-5l6tj\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-dshm\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050868 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-home\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050935 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050917    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tmp-dir\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.050988 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.050951    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-model-cache\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.052614 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.052597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-dshm\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.052841 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.052824    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tls-certs\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.059050 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.059027    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6tj\" (UniqueName: \"kubernetes.io/projected/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-kube-api-access-5l6tj\") pod \"test-model-05ecbf-kserve-7f7f5dd96c-25t2v\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") " pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.100217 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.100193    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:08.367975 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.367953    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"]
Sep 07 10:26:08.369792 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:26:08.369765    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96d3ec45_7be2_4240_bdbd_8b1fc5cfcdfc.slice/crio-12d478b2dd90b594fc473d28d3bbf1b0412e0adc4d90a87314641d86ad80f25b WatchSource:0}: Error finding container 12d478b2dd90b594fc473d28d3bbf1b0412e0adc4d90a87314641d86ad80f25b: Status 404 returned error can't find the container with id 12d478b2dd90b594fc473d28d3bbf1b0412e0adc4d90a87314641d86ad80f25b
Sep 07 10:26:08.776938 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.776850    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v" event={"ID":"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc","Type":"ContainerStarted","Data":"5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174"}
Sep 07 10:26:08.776938 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.776893    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v" event={"ID":"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc","Type":"ContainerStarted","Data":"12d478b2dd90b594fc473d28d3bbf1b0412e0adc4d90a87314641d86ad80f25b"}
Sep 07 10:26:08.795844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:08.795791    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v" podStartSLOduration=1.795776341 podStartE2EDuration="1.795776341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:08.793216709 +0000 UTC m=+875.341731696" watchObservedRunningTime="2026-09-07 10:26:08.795776341 +0000 UTC m=+875.344291329"
Sep 07 10:26:10.294989 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:10.294945    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"]
Sep 07 10:26:10.311745 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:10.311698    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:10.893101 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:10.893047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:12.414375 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:12.414334    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:12.688427 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:12.688336    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:12.921723 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:12.921688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:13.632309 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:13.632273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:14.415682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:14.415640    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:18.100532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:18.100485    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:18.100961 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:18.100546    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:18.112986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:18.112961    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:18.836172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:18.836142    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:26:18.920495 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:18.920462    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:19.295409 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.295375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:26:19.654682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.654562    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:19.663493 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.663455    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"]
Sep 07 10:26:19.663815 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.663783    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r" podUID="7cd066e6-eff6-4709-b668-d7e02bf79808" containerName="payload-processing" containerID="cri-o://63fe6018e6d6d2d6c97f602ca5134c12956f3679446debacc8d76a1d656d216b" gracePeriod=30
Sep 07 10:26:19.726413 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.726375    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:19.832315 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.832278    2560 generic.go:356] "Generic (PLEG): container finished" podID="7cd066e6-eff6-4709-b668-d7e02bf79808" containerID="63fe6018e6d6d2d6c97f602ca5134c12956f3679446debacc8d76a1d656d216b" exitCode=0
Sep 07 10:26:19.832477 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.832335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r" event={"ID":"7cd066e6-eff6-4709-b668-d7e02bf79808","Type":"ContainerDied","Data":"63fe6018e6d6d2d6c97f602ca5134c12956f3679446debacc8d76a1d656d216b"}
Sep 07 10:26:19.856571 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.856485    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:19.927896 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.927868    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:26:19.952580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:19.952524    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:20.066850 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.066821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd066e6-eff6-4709-b668-d7e02bf79808-plugins-config-volume\") pod \"7cd066e6-eff6-4709-b668-d7e02bf79808\" (UID: \"7cd066e6-eff6-4709-b668-d7e02bf79808\") "
Sep 07 10:26:20.067035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.066911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddghw\" (UniqueName: \"kubernetes.io/projected/7cd066e6-eff6-4709-b668-d7e02bf79808-kube-api-access-ddghw\") pod \"7cd066e6-eff6-4709-b668-d7e02bf79808\" (UID: \"7cd066e6-eff6-4709-b668-d7e02bf79808\") "
Sep 07 10:26:20.067296 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.067269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd066e6-eff6-4709-b668-d7e02bf79808-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cd066e6-eff6-4709-b668-d7e02bf79808" (UID: "7cd066e6-eff6-4709-b668-d7e02bf79808"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:26:20.068924 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.068902    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd066e6-eff6-4709-b668-d7e02bf79808-kube-api-access-ddghw" (OuterVolumeSpecName: "kube-api-access-ddghw") pod "7cd066e6-eff6-4709-b668-d7e02bf79808" (UID: "7cd066e6-eff6-4709-b668-d7e02bf79808"). InnerVolumeSpecName "kube-api-access-ddghw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:20.168276 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.168198    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd066e6-eff6-4709-b668-d7e02bf79808-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:20.168276 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.168226    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddghw\" (UniqueName: \"kubernetes.io/projected/7cd066e6-eff6-4709-b668-d7e02bf79808-kube-api-access-ddghw\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:20.838430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.838401    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"
Sep 07 10:26:20.838897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.838398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r" event={"ID":"7cd066e6-eff6-4709-b668-d7e02bf79808","Type":"ContainerDied","Data":"65cd1e73382e34dc3140b70cc1a9f209c24a8df442bbeec3c59e733633fcf461"}
Sep 07 10:26:20.838897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.838536    2560 scope.go:117] "RemoveContainer" containerID="63fe6018e6d6d2d6c97f602ca5134c12956f3679446debacc8d76a1d656d216b"
Sep 07 10:26:20.876700 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.876667    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"]
Sep 07 10:26:20.878489 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.878466    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-31186eae-f5d9dd75-j795r"]
Sep 07 10:26:20.931995 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:20.931965    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:21.343475 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:21.343431    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:21.631936 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:21.631858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:21.836598 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:21.836567    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:21.894508 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:21.894440    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd066e6-eff6-4709-b668-d7e02bf79808" path="/var/lib/kubelet/pods/7cd066e6-eff6-4709-b668-d7e02bf79808/volumes"
Sep 07 10:26:23.035584 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.035549    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:23.612552 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.612518    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"]
Sep 07 10:26:23.612763 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.612738    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj" podUID="44ac8c84-920a-4f74-acd0-2f8658a1c7d6" containerName="maas-api" containerID="cri-o://304a52ec7dd80a9c56e4851b1b9149c6080ec5a08318654acd4c124f39bc7de4" gracePeriod=30
Sep 07 10:26:23.665570 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.665536    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"]
Sep 07 10:26:23.665836 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.665805    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx" podUID="5cf0d337-3048-4480-8f28-09fca608aab0" containerName="payload-processing" containerID="cri-o://4cd4c5d6a86c88b18f70aebd678171f649cad372a220b7bece51f1b4bc7885a1" gracePeriod=30
Sep 07 10:26:23.696633 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.696604    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:23.770312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.770273    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:23.855855 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.855814    2560 generic.go:356] "Generic (PLEG): container finished" podID="44ac8c84-920a-4f74-acd0-2f8658a1c7d6" containerID="304a52ec7dd80a9c56e4851b1b9149c6080ec5a08318654acd4c124f39bc7de4" exitCode=0
Sep 07 10:26:23.855987 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.855916    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj" event={"ID":"44ac8c84-920a-4f74-acd0-2f8658a1c7d6","Type":"ContainerDied","Data":"304a52ec7dd80a9c56e4851b1b9149c6080ec5a08318654acd4c124f39bc7de4"}
Sep 07 10:26:23.857873 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.857838    2560 generic.go:356] "Generic (PLEG): container finished" podID="5cf0d337-3048-4480-8f28-09fca608aab0" containerID="4cd4c5d6a86c88b18f70aebd678171f649cad372a220b7bece51f1b4bc7885a1" exitCode=0
Sep 07 10:26:23.858006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.857924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx" event={"ID":"5cf0d337-3048-4480-8f28-09fca608aab0","Type":"ContainerDied","Data":"4cd4c5d6a86c88b18f70aebd678171f649cad372a220b7bece51f1b4bc7885a1"}
Sep 07 10:26:23.861876 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.861850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:23.918736 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.918711    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:26:23.929944 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.929924    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:26:23.942199 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:23.942169    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:24.002175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.002139    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf0d337-3048-4480-8f28-09fca608aab0-plugins-config-volume\") pod \"5cf0d337-3048-4480-8f28-09fca608aab0\" (UID: \"5cf0d337-3048-4480-8f28-09fca608aab0\") "
Sep 07 10:26:24.002175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.002183    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-maas-api-tls\") pod \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") "
Sep 07 10:26:24.002421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.002282    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7562\" (UniqueName: \"kubernetes.io/projected/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-kube-api-access-v7562\") pod \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") "
Sep 07 10:26:24.002421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.002302    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-openshift-service-ca\") pod \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") "
Sep 07 10:26:24.002421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.002320    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq87j\" (UniqueName: \"kubernetes.io/projected/5cf0d337-3048-4480-8f28-09fca608aab0-kube-api-access-zq87j\") pod \"5cf0d337-3048-4480-8f28-09fca608aab0\" (UID: \"5cf0d337-3048-4480-8f28-09fca608aab0\") "
Sep 07 10:26:24.002421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.002361    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-metrics-certs\") pod \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\" (UID: \"44ac8c84-920a-4f74-acd0-2f8658a1c7d6\") "
Sep 07 10:26:24.002660 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.002630    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf0d337-3048-4480-8f28-09fca608aab0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cf0d337-3048-4480-8f28-09fca608aab0" (UID: "5cf0d337-3048-4480-8f28-09fca608aab0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:26:24.002725 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.002674    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44ac8c84-920a-4f74-acd0-2f8658a1c7d6" (UID: "44ac8c84-920a-4f74-acd0-2f8658a1c7d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:26:24.004656 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.004607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf0d337-3048-4480-8f28-09fca608aab0-kube-api-access-zq87j" (OuterVolumeSpecName: "kube-api-access-zq87j") pod "5cf0d337-3048-4480-8f28-09fca608aab0" (UID: "5cf0d337-3048-4480-8f28-09fca608aab0"). InnerVolumeSpecName "kube-api-access-zq87j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:24.004790 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.004673    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "44ac8c84-920a-4f74-acd0-2f8658a1c7d6" (UID: "44ac8c84-920a-4f74-acd0-2f8658a1c7d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:26:24.004790 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.004697    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-kube-api-access-v7562" (OuterVolumeSpecName: "kube-api-access-v7562") pod "44ac8c84-920a-4f74-acd0-2f8658a1c7d6" (UID: "44ac8c84-920a-4f74-acd0-2f8658a1c7d6"). InnerVolumeSpecName "kube-api-access-v7562". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:24.004790 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.004701    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44ac8c84-920a-4f74-acd0-2f8658a1c7d6" (UID: "44ac8c84-920a-4f74-acd0-2f8658a1c7d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:26:24.103503 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.103467    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-maas-api-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:24.103503 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.103498    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7562\" (UniqueName: \"kubernetes.io/projected/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-kube-api-access-v7562\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:24.103503 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.103512    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-openshift-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:24.103943 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.103525    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq87j\" (UniqueName: \"kubernetes.io/projected/5cf0d337-3048-4480-8f28-09fca608aab0-kube-api-access-zq87j\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:24.103943 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.103536    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44ac8c84-920a-4f74-acd0-2f8658a1c7d6-metrics-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:24.103943 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.103547    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cf0d337-3048-4480-8f28-09fca608aab0-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:24.228188 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.228099    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:24.863779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.863750    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"
Sep 07 10:26:24.863779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.863763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj" event={"ID":"44ac8c84-920a-4f74-acd0-2f8658a1c7d6","Type":"ContainerDied","Data":"933e576beabc58587dc1794295b68766b8bfcfdeff260a01b7ab6229f06b8e42"}
Sep 07 10:26:24.864017 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.863821    2560 scope.go:117] "RemoveContainer" containerID="304a52ec7dd80a9c56e4851b1b9149c6080ec5a08318654acd4c124f39bc7de4"
Sep 07 10:26:24.865098 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.865033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx" event={"ID":"5cf0d337-3048-4480-8f28-09fca608aab0","Type":"ContainerDied","Data":"4b5e6821795bcdc6b802c133c80afe8df5a7bfd182618dab4867b2ef6ab63a57"}
Sep 07 10:26:24.865098 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.865059    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"
Sep 07 10:26:24.872596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.872528    2560 scope.go:117] "RemoveContainer" containerID="4cd4c5d6a86c88b18f70aebd678171f649cad372a220b7bece51f1b4bc7885a1"
Sep 07 10:26:24.895105 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.895080    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"]
Sep 07 10:26:24.902715 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.902693    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ff5470-5cb85d499d-x5vxj"]
Sep 07 10:26:24.914609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.914571    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"]
Sep 07 10:26:24.917643 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.917619    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ff5470-67cfdcfc89-tqgxx"]
Sep 07 10:26:24.945222 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:24.945196    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:25.048256 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.048221    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:25.315677 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.315641    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k" podUID="6d548bca-247f-4908-886e-14e4395acb66" containerName="main" containerID="cri-o://3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878" gracePeriod=45
Sep 07 10:26:25.413151 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.413112    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:25.514690 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.514647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:25.522159 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522128    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"]
Sep 07 10:26:25.522498 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522485    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd066e6-eff6-4709-b668-d7e02bf79808" containerName="payload-processing"
Sep 07 10:26:25.522547 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522500    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd066e6-eff6-4709-b668-d7e02bf79808" containerName="payload-processing"
Sep 07 10:26:25.522547 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522517    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf0d337-3048-4480-8f28-09fca608aab0" containerName="payload-processing"
Sep 07 10:26:25.522547 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522522    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf0d337-3048-4480-8f28-09fca608aab0" containerName="payload-processing"
Sep 07 10:26:25.522547 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522538    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ac8c84-920a-4f74-acd0-2f8658a1c7d6" containerName="maas-api"
Sep 07 10:26:25.522547 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522547    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ac8c84-920a-4f74-acd0-2f8658a1c7d6" containerName="maas-api"
Sep 07 10:26:25.522703 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522602    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd066e6-eff6-4709-b668-d7e02bf79808" containerName="payload-processing"
Sep 07 10:26:25.522703 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522611    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cf0d337-3048-4480-8f28-09fca608aab0" containerName="payload-processing"
Sep 07 10:26:25.522703 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.522621    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ac8c84-920a-4f74-acd0-2f8658a1c7d6" containerName="maas-api"
Sep 07 10:26:25.525470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.525455    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:25.528008 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.527985    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-39223a0b-dockercfg-z96jt\""
Sep 07 10:26:25.528136 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.528015    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-39223a0b\""
Sep 07 10:26:25.534189 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.534165    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"]
Sep 07 10:26:25.568059 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.567998    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:26:25.616666 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.616633    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d548bca-247f-4908-886e-14e4395acb66-tls-certs\") pod \"6d548bca-247f-4908-886e-14e4395acb66\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") "
Sep 07 10:26:25.616830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.616679    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-dshm\") pod \"6d548bca-247f-4908-886e-14e4395acb66\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") "
Sep 07 10:26:25.616830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.616699    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-model-cache\") pod \"6d548bca-247f-4908-886e-14e4395acb66\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") "
Sep 07 10:26:25.616830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.616718    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7kfm\" (UniqueName: \"kubernetes.io/projected/6d548bca-247f-4908-886e-14e4395acb66-kube-api-access-k7kfm\") pod \"6d548bca-247f-4908-886e-14e4395acb66\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") "
Sep 07 10:26:25.616830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.616746    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-tmp-dir\") pod \"6d548bca-247f-4908-886e-14e4395acb66\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") "
Sep 07 10:26:25.616830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.616764    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-home\") pod \"6d548bca-247f-4908-886e-14e4395acb66\" (UID: \"6d548bca-247f-4908-886e-14e4395acb66\") "
Sep 07 10:26:25.617160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.616952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7v2v\" (UniqueName: \"kubernetes.io/projected/8ff59943-d4b7-40c6-a71f-5ada380495a5-kube-api-access-q7v2v\") pod \"payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j\" (UID: \"8ff59943-d4b7-40c6-a71f-5ada380495a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:25.617160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.617013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff59943-d4b7-40c6-a71f-5ada380495a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j\" (UID: \"8ff59943-d4b7-40c6-a71f-5ada380495a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:25.617160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.617019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d548bca-247f-4908-886e-14e4395acb66" (UID: "6d548bca-247f-4908-886e-14e4395acb66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:26:25.617160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.617035    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d548bca-247f-4908-886e-14e4395acb66" (UID: "6d548bca-247f-4908-886e-14e4395acb66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:26:25.617372 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.617168    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-home" (OuterVolumeSpecName: "home") pod "6d548bca-247f-4908-886e-14e4395acb66" (UID: "6d548bca-247f-4908-886e-14e4395acb66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:26:25.617372 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.617319    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-model-cache\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:25.617372 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.617342    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-tmp-dir\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:25.617372 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.617355    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-home\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:25.618930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.618897    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-dshm" (OuterVolumeSpecName: "dshm") pod "6d548bca-247f-4908-886e-14e4395acb66" (UID: "6d548bca-247f-4908-886e-14e4395acb66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:26:25.619042 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.618962    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d548bca-247f-4908-886e-14e4395acb66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d548bca-247f-4908-886e-14e4395acb66" (UID: "6d548bca-247f-4908-886e-14e4395acb66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:26:25.619042 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.618986    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d548bca-247f-4908-886e-14e4395acb66-kube-api-access-k7kfm" (OuterVolumeSpecName: "kube-api-access-k7kfm") pod "6d548bca-247f-4908-886e-14e4395acb66" (UID: "6d548bca-247f-4908-886e-14e4395acb66"). InnerVolumeSpecName "kube-api-access-k7kfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:25.718754 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.718712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7v2v\" (UniqueName: \"kubernetes.io/projected/8ff59943-d4b7-40c6-a71f-5ada380495a5-kube-api-access-q7v2v\") pod \"payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j\" (UID: \"8ff59943-d4b7-40c6-a71f-5ada380495a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:25.718949 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.718764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff59943-d4b7-40c6-a71f-5ada380495a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j\" (UID: \"8ff59943-d4b7-40c6-a71f-5ada380495a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:25.718949 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.718897    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kfm\" (UniqueName: \"kubernetes.io/projected/6d548bca-247f-4908-886e-14e4395acb66-kube-api-access-k7kfm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:25.718949 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.718916    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d548bca-247f-4908-886e-14e4395acb66-tls-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:25.718949 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.718931    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d548bca-247f-4908-886e-14e4395acb66-dshm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:25.719522 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.719502    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff59943-d4b7-40c6-a71f-5ada380495a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j\" (UID: \"8ff59943-d4b7-40c6-a71f-5ada380495a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:25.727320 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.727295    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7v2v\" (UniqueName: \"kubernetes.io/projected/8ff59943-d4b7-40c6-a71f-5ada380495a5-kube-api-access-q7v2v\") pod \"payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j\" (UID: \"8ff59943-d4b7-40c6-a71f-5ada380495a5\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:25.837148 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.837041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:25.875185 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.875147    2560 generic.go:356] "Generic (PLEG): container finished" podID="6d548bca-247f-4908-886e-14e4395acb66" containerID="3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878" exitCode=0
Sep 07 10:26:25.875413 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.875230    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"
Sep 07 10:26:25.875413 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.875227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k" event={"ID":"6d548bca-247f-4908-886e-14e4395acb66","Type":"ContainerDied","Data":"3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878"}
Sep 07 10:26:25.875413 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.875362    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k" event={"ID":"6d548bca-247f-4908-886e-14e4395acb66","Type":"ContainerDied","Data":"135e651d0ea68e705b289649067a2f9cfbaaeef39f97f8002545c4396adc42fd"}
Sep 07 10:26:25.875413 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.875385    2560 scope.go:117] "RemoveContainer" containerID="3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878"
Sep 07 10:26:25.885540 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.885306    2560 scope.go:117] "RemoveContainer" containerID="3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878"
Sep 07 10:26:25.885805 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:26:25.885717    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878\": container with ID starting with 3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878 not found: ID does not exist" containerID="3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878"
Sep 07 10:26:25.885805 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.885752    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878"} err="failed to get container status \"3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878\": rpc error: code = NotFound desc = could not find container \"3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878\": container with ID starting with 3bf30722cb437ce703d4ea109a25b49f5168830d969c1ed0a45e6f839052b878 not found: ID does not exist"
Sep 07 10:26:25.897086 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.897041    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ac8c84-920a-4f74-acd0-2f8658a1c7d6" path="/var/lib/kubelet/pods/44ac8c84-920a-4f74-acd0-2f8658a1c7d6/volumes"
Sep 07 10:26:25.897651 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.897625    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf0d337-3048-4480-8f28-09fca608aab0" path="/var/lib/kubelet/pods/5cf0d337-3048-4480-8f28-09fca608aab0/volumes"
Sep 07 10:26:25.901548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.901494    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"]
Sep 07 10:26:25.904268 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:25.904199    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-5bc2bc81-kserve-5886fdd7f5-d5z8k"]
Sep 07 10:26:26.103865 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:26.103793    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:26.117309 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:26.117290    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"]
Sep 07 10:26:26.118967 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:26:26.118942    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff59943_d4b7_40c6_a71f_5ada380495a5.slice/crio-8c2787ac99ea06d806021c85950042d780af38ae936e337e48d41cef3d645adf WatchSource:0}: Error finding container 8c2787ac99ea06d806021c85950042d780af38ae936e337e48d41cef3d645adf: Status 404 returned error can't find the container with id 8c2787ac99ea06d806021c85950042d780af38ae936e337e48d41cef3d645adf
Sep 07 10:26:26.881966 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:26.881930    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j" event={"ID":"8ff59943-d4b7-40c6-a71f-5ada380495a5","Type":"ContainerStarted","Data":"15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602"}
Sep 07 10:26:26.882616 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:26.881977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j" event={"ID":"8ff59943-d4b7-40c6-a71f-5ada380495a5","Type":"ContainerStarted","Data":"8c2787ac99ea06d806021c85950042d780af38ae936e337e48d41cef3d645adf"}
Sep 07 10:26:26.882616 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:26.882084    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:26.897917 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:26.897869    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j" podStartSLOduration=1.897853719 podStartE2EDuration="1.897853719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:26.897373804 +0000 UTC m=+893.445888792" watchObservedRunningTime="2026-09-07 10:26:26.897853719 +0000 UTC m=+893.446368706"
Sep 07 10:26:27.895173 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:27.895143    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d548bca-247f-4908-886e-14e4395acb66" path="/var/lib/kubelet/pods/6d548bca-247f-4908-886e-14e4395acb66/volumes"
Sep 07 10:26:30.060020 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:30.059976    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:30.160932 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:30.160893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:30.330671 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:30.330575    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:30.781532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:30.781438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:31.086507 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:31.086476    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:31.187462 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:31.187431    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:33.720887 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:33.720854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:34.334958 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:34.334915    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:34.435455 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:34.435420    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:35.433847 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.433814    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"]
Sep 07 10:26:35.434404 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.434195    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d548bca-247f-4908-886e-14e4395acb66" containerName="main"
Sep 07 10:26:35.434404 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.434207    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d548bca-247f-4908-886e-14e4395acb66" containerName="main"
Sep 07 10:26:35.434404 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.434258    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d548bca-247f-4908-886e-14e4395acb66" containerName="main"
Sep 07 10:26:35.438645 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.438621    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.440986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.440964    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-81af70-serving-cert\""
Sep 07 10:26:35.449245 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.449227    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"]
Sep 07 10:26:35.478322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.478287    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:35.505262 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.505236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4c44fa-1851-45f7-a60b-3237c4ca9196-openshift-service-ca\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.505353 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.505289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-metrics-certs\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.505353 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.505334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.505430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.505352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkp9j\" (UniqueName: \"kubernetes.io/projected/4d4c44fa-1851-45f7-a60b-3237c4ca9196-kube-api-access-qkp9j\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.538044 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.538016    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:35.606229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.606201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4c44fa-1851-45f7-a60b-3237c4ca9196-openshift-service-ca\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.606371 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.606249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-metrics-certs\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.606371 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.606273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.606371 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.606290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkp9j\" (UniqueName: \"kubernetes.io/projected/4d4c44fa-1851-45f7-a60b-3237c4ca9196-kube-api-access-qkp9j\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.606575 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:26:35.606443    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-serving-cert: secret "maas-api-e2e-explicit-81af70-serving-cert" not found
Sep 07 10:26:35.606575 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:26:35.606509    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls podName:4d4c44fa-1851-45f7-a60b-3237c4ca9196 nodeName:}" failed. No retries permitted until 2026-09-07 10:26:36.106491522 +0000 UTC m=+902.655006490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls") pod "maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x" (UID: "4d4c44fa-1851-45f7-a60b-3237c4ca9196") : secret "maas-api-e2e-explicit-81af70-serving-cert" not found
Sep 07 10:26:35.606805 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.606785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4c44fa-1851-45f7-a60b-3237c4ca9196-openshift-service-ca\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.608701 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.608680    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-metrics-certs\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.614290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.614269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkp9j\" (UniqueName: \"kubernetes.io/projected/4d4c44fa-1851-45f7-a60b-3237c4ca9196-kube-api-access-qkp9j\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:35.943531 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:35.943501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:36.058854 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.058821    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:36.111714 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.111686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:36.113981 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.113954    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls\") pod \"maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:36.118223 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.118199    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"]
Sep 07 10:26:36.121801 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.121785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.124509 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.124489    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-81af70\""
Sep 07 10:26:36.141494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.141468    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"]
Sep 07 10:26:36.201856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.201791    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:36.212563 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.212540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d69m6\" (UniqueName: \"kubernetes.io/projected/740ef2d2-baad-4e33-8335-ce68d166aa4b-kube-api-access-d69m6\") pod \"payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p\" (UID: \"740ef2d2-baad-4e33-8335-ce68d166aa4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.212790 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.212760    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ef2d2-baad-4e33-8335-ce68d166aa4b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p\" (UID: \"740ef2d2-baad-4e33-8335-ce68d166aa4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.313786 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.313761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69m6\" (UniqueName: \"kubernetes.io/projected/740ef2d2-baad-4e33-8335-ce68d166aa4b-kube-api-access-d69m6\") pod \"payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p\" (UID: \"740ef2d2-baad-4e33-8335-ce68d166aa4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.313901 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.313808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ef2d2-baad-4e33-8335-ce68d166aa4b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p\" (UID: \"740ef2d2-baad-4e33-8335-ce68d166aa4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.314501 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.314483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ef2d2-baad-4e33-8335-ce68d166aa4b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p\" (UID: \"740ef2d2-baad-4e33-8335-ce68d166aa4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.321535 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.321508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69m6\" (UniqueName: \"kubernetes.io/projected/740ef2d2-baad-4e33-8335-ce68d166aa4b-kube-api-access-d69m6\") pod \"payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p\" (UID: \"740ef2d2-baad-4e33-8335-ce68d166aa4b\") " pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.351137 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.351114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:36.432210 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.432178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.618465 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.618442    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"]
Sep 07 10:26:36.620272 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:26:36.620245    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4c44fa_1851_45f7_a60b_3237c4ca9196.slice/crio-d2b26bec567a4cc736d65ed8c70672f8566f2a79f35dbc039bab046d4b111e84 WatchSource:0}: Error finding container d2b26bec567a4cc736d65ed8c70672f8566f2a79f35dbc039bab046d4b111e84: Status 404 returned error can't find the container with id d2b26bec567a4cc736d65ed8c70672f8566f2a79f35dbc039bab046d4b111e84
Sep 07 10:26:36.699688 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.699665    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"]
Sep 07 10:26:36.701379 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:26:36.701348    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740ef2d2_baad_4e33_8335_ce68d166aa4b.slice/crio-fc01bd0de7af62ddfa61e284194c093fd6631213c8c2889ea6686e373fc9ce9b WatchSource:0}: Error finding container fc01bd0de7af62ddfa61e284194c093fd6631213c8c2889ea6686e373fc9ce9b: Status 404 returned error can't find the container with id fc01bd0de7af62ddfa61e284194c093fd6631213c8c2889ea6686e373fc9ce9b
Sep 07 10:26:36.929634 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.929602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p" event={"ID":"740ef2d2-baad-4e33-8335-ce68d166aa4b","Type":"ContainerStarted","Data":"a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f"}
Sep 07 10:26:36.929813 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.929652    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p" event={"ID":"740ef2d2-baad-4e33-8335-ce68d166aa4b","Type":"ContainerStarted","Data":"fc01bd0de7af62ddfa61e284194c093fd6631213c8c2889ea6686e373fc9ce9b"}
Sep 07 10:26:36.929813 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.929693    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:36.930861 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.930838    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x" event={"ID":"4d4c44fa-1851-45f7-a60b-3237c4ca9196","Type":"ContainerStarted","Data":"d2b26bec567a4cc736d65ed8c70672f8566f2a79f35dbc039bab046d4b111e84"}
Sep 07 10:26:36.945902 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:36.945856    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p" podStartSLOduration=0.945843648 podStartE2EDuration="945.843648ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:36.944438613 +0000 UTC m=+903.492953660" watchObservedRunningTime="2026-09-07 10:26:36.945843648 +0000 UTC m=+903.494358698"
Sep 07 10:26:37.895039 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:37.895002    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:37.938045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:37.938012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x" event={"ID":"4d4c44fa-1851-45f7-a60b-3237c4ca9196","Type":"ContainerStarted","Data":"9e9ad46eb759114cac652b1b8c59004d473ec7b943e5425d902f176c4a810caa"}
Sep 07 10:26:37.938217 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:37.938114    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:37.956629 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:37.956582    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x" podStartSLOduration=2.477537629 podStartE2EDuration="2.956566812s" totalImagesPullingTime="479.029183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:26:37.955437516 +0000 UTC m=+904.503952504" watchObservedRunningTime="2026-09-07 10:26:37.956566812 +0000 UTC m=+904.505081798"
Sep 07 10:26:39.091748 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:39.091715    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:26:40.677980 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:40.677953    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:43.948839 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:43.948807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:26:45.881013 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:45.880967    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:47.940947 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:47.940916    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:26:55.544312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.544276    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:55.582207 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.582173    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"]
Sep 07 10:26:55.582468 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.582415    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j" podUID="8ff59943-d4b7-40c6-a71f-5ada380495a5" containerName="payload-pre-processing" containerID="cri-o://15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602" gracePeriod=30
Sep 07 10:26:55.645205 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.645173    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:55.745587 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.745396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:55.805129 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.805012    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:55.826589 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.826563    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:55.914973 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.914935    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:55.991747 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.991709    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7v2v\" (UniqueName: \"kubernetes.io/projected/8ff59943-d4b7-40c6-a71f-5ada380495a5-kube-api-access-q7v2v\") pod \"8ff59943-d4b7-40c6-a71f-5ada380495a5\" (UID: \"8ff59943-d4b7-40c6-a71f-5ada380495a5\") "
Sep 07 10:26:55.991981 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.991797    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff59943-d4b7-40c6-a71f-5ada380495a5-plugins-config-volume\") pod \"8ff59943-d4b7-40c6-a71f-5ada380495a5\" (UID: \"8ff59943-d4b7-40c6-a71f-5ada380495a5\") "
Sep 07 10:26:55.992180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.992155    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff59943-d4b7-40c6-a71f-5ada380495a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ff59943-d4b7-40c6-a71f-5ada380495a5" (UID: "8ff59943-d4b7-40c6-a71f-5ada380495a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:26:55.993786 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:55.993763    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff59943-d4b7-40c6-a71f-5ada380495a5-kube-api-access-q7v2v" (OuterVolumeSpecName: "kube-api-access-q7v2v") pod "8ff59943-d4b7-40c6-a71f-5ada380495a5" (UID: "8ff59943-d4b7-40c6-a71f-5ada380495a5"). InnerVolumeSpecName "kube-api-access-q7v2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:26:56.020039 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.020007    2560 generic.go:356] "Generic (PLEG): container finished" podID="8ff59943-d4b7-40c6-a71f-5ada380495a5" containerID="15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602" exitCode=0
Sep 07 10:26:56.020187 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.020059    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"
Sep 07 10:26:56.020187 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.020095    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j" event={"ID":"8ff59943-d4b7-40c6-a71f-5ada380495a5","Type":"ContainerDied","Data":"15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602"}
Sep 07 10:26:56.020187 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.020173    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j" event={"ID":"8ff59943-d4b7-40c6-a71f-5ada380495a5","Type":"ContainerDied","Data":"8c2787ac99ea06d806021c85950042d780af38ae936e337e48d41cef3d645adf"}
Sep 07 10:26:56.020310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.020199    2560 scope.go:117] "RemoveContainer" containerID="15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602"
Sep 07 10:26:56.029250 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.029230    2560 scope.go:117] "RemoveContainer" containerID="15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602"
Sep 07 10:26:56.029559 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:26:56.029540    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602\": container with ID starting with 15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602 not found: ID does not exist" containerID="15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602"
Sep 07 10:26:56.029633 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.029574    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602"} err="failed to get container status \"15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602\": rpc error: code = NotFound desc = could not find container \"15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602\": container with ID starting with 15b2f0bb6bf3051749b492b30321307763ac431f8e173782796a955e43f51602 not found: ID does not exist"
Sep 07 10:26:56.040791 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.040767    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"]
Sep 07 10:26:56.044893 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.044872    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-39223a0b-fd5798758-t2f6j"]
Sep 07 10:26:56.092907 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.092844    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7v2v\" (UniqueName: \"kubernetes.io/projected/8ff59943-d4b7-40c6-a71f-5ada380495a5-kube-api-access-q7v2v\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:56.092907 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.092868    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff59943-d4b7-40c6-a71f-5ada380495a5-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:26:56.434691 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.434606    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:56.571759 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.571727    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:56.637989 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.637955    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:56.708902 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:56.708802    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:57.894998 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:57.894966    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ff59943-d4b7-40c6-a71f-5ada380495a5" path="/var/lib/kubelet/pods/8ff59943-d4b7-40c6-a71f-5ada380495a5/volumes"
Sep 07 10:26:58.062601 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:58.062563    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:59.227392 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:59.227358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:26:59.786356 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:26:59.786315    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:27:03.625420 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:03.625382    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:03.697858 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:03.697832    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:03.781556 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:03.781529    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:04.289668 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:04.289625    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:04.389831 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:04.389799    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:14.792257 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:14.792224    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:27:23.524962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:23.524919    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:27.392642 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:27.392608    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:29.795722 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:29.795678    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:27:30.170587 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:30.170501    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:35.916172 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:35.916137    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:35.975801 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:35.975775    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:36.053057 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:36.053022    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:36.153050 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:36.153018    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:38.101184 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.101155    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"]
Sep 07 10:27:38.101603 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.101399    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x" podUID="4d4c44fa-1851-45f7-a60b-3237c4ca9196" containerName="maas-api" containerID="cri-o://9e9ad46eb759114cac652b1b8c59004d473ec7b943e5425d902f176c4a810caa" gracePeriod=30
Sep 07 10:27:38.125168 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.125144    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:38.148806 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.148778    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"]
Sep 07 10:27:38.149018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.148981    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p" podUID="740ef2d2-baad-4e33-8335-ce68d166aa4b" containerName="payload-processing" containerID="cri-o://a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f" gracePeriod=30
Sep 07 10:27:38.193596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.193570    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:38.227413 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.227390    2560 generic.go:356] "Generic (PLEG): container finished" podID="4d4c44fa-1851-45f7-a60b-3237c4ca9196" containerID="9e9ad46eb759114cac652b1b8c59004d473ec7b943e5425d902f176c4a810caa" exitCode=0
Sep 07 10:27:38.227526 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.227463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x" event={"ID":"4d4c44fa-1851-45f7-a60b-3237c4ca9196","Type":"ContainerDied","Data":"9e9ad46eb759114cac652b1b8c59004d473ec7b943e5425d902f176c4a810caa"}
Sep 07 10:27:38.293926 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.293862    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:38.355144 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.355054    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:38.396534 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.396514    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:27:38.426304 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.426285    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:27:38.541195 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.541161    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4c44fa-1851-45f7-a60b-3237c4ca9196-openshift-service-ca\") pod \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") "
Sep 07 10:27:38.541195 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.541193    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d69m6\" (UniqueName: \"kubernetes.io/projected/740ef2d2-baad-4e33-8335-ce68d166aa4b-kube-api-access-d69m6\") pod \"740ef2d2-baad-4e33-8335-ce68d166aa4b\" (UID: \"740ef2d2-baad-4e33-8335-ce68d166aa4b\") "
Sep 07 10:27:38.541412 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.541249    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-metrics-certs\") pod \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") "
Sep 07 10:27:38.541412 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.541291    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls\") pod \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") "
Sep 07 10:27:38.541412 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.541320    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkp9j\" (UniqueName: \"kubernetes.io/projected/4d4c44fa-1851-45f7-a60b-3237c4ca9196-kube-api-access-qkp9j\") pod \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\" (UID: \"4d4c44fa-1851-45f7-a60b-3237c4ca9196\") "
Sep 07 10:27:38.541412 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.541346    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ef2d2-baad-4e33-8335-ce68d166aa4b-plugins-config-volume\") pod \"740ef2d2-baad-4e33-8335-ce68d166aa4b\" (UID: \"740ef2d2-baad-4e33-8335-ce68d166aa4b\") "
Sep 07 10:27:38.541701 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.541566    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4c44fa-1851-45f7-a60b-3237c4ca9196-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d4c44fa-1851-45f7-a60b-3237c4ca9196" (UID: "4d4c44fa-1851-45f7-a60b-3237c4ca9196"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:27:38.541830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.541779    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740ef2d2-baad-4e33-8335-ce68d166aa4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "740ef2d2-baad-4e33-8335-ce68d166aa4b" (UID: "740ef2d2-baad-4e33-8335-ce68d166aa4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:27:38.543529 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.543504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d4c44fa-1851-45f7-a60b-3237c4ca9196" (UID: "4d4c44fa-1851-45f7-a60b-3237c4ca9196"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:27:38.543529 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.543518    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740ef2d2-baad-4e33-8335-ce68d166aa4b-kube-api-access-d69m6" (OuterVolumeSpecName: "kube-api-access-d69m6") pod "740ef2d2-baad-4e33-8335-ce68d166aa4b" (UID: "740ef2d2-baad-4e33-8335-ce68d166aa4b"). InnerVolumeSpecName "kube-api-access-d69m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:27:38.543673 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.543542    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4d4c44fa-1851-45f7-a60b-3237c4ca9196" (UID: "4d4c44fa-1851-45f7-a60b-3237c4ca9196"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:27:38.543673 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.543549    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4c44fa-1851-45f7-a60b-3237c4ca9196-kube-api-access-qkp9j" (OuterVolumeSpecName: "kube-api-access-qkp9j") pod "4d4c44fa-1851-45f7-a60b-3237c4ca9196" (UID: "4d4c44fa-1851-45f7-a60b-3237c4ca9196"). InnerVolumeSpecName "kube-api-access-qkp9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:27:38.642251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.642163    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d4c44fa-1851-45f7-a60b-3237c4ca9196-openshift-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:27:38.642251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.642193    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d69m6\" (UniqueName: \"kubernetes.io/projected/740ef2d2-baad-4e33-8335-ce68d166aa4b-kube-api-access-d69m6\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:27:38.642251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.642209    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-metrics-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:27:38.642251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.642220    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d4c44fa-1851-45f7-a60b-3237c4ca9196-maas-api-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:27:38.642251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.642231    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkp9j\" (UniqueName: \"kubernetes.io/projected/4d4c44fa-1851-45f7-a60b-3237c4ca9196-kube-api-access-qkp9j\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:27:38.642251 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.642243    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/740ef2d2-baad-4e33-8335-ce68d166aa4b-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:27:38.655506 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:38.655477    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:39.233681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.233649    2560 generic.go:356] "Generic (PLEG): container finished" podID="740ef2d2-baad-4e33-8335-ce68d166aa4b" containerID="a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f" exitCode=0
Sep 07 10:27:39.234138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.233693    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"
Sep 07 10:27:39.234138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.233726    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p" event={"ID":"740ef2d2-baad-4e33-8335-ce68d166aa4b","Type":"ContainerDied","Data":"a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f"}
Sep 07 10:27:39.234138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.233772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p" event={"ID":"740ef2d2-baad-4e33-8335-ce68d166aa4b","Type":"ContainerDied","Data":"fc01bd0de7af62ddfa61e284194c093fd6631213c8c2889ea6686e373fc9ce9b"}
Sep 07 10:27:39.234138 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.233795    2560 scope.go:117] "RemoveContainer" containerID="a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f"
Sep 07 10:27:39.235189 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.235174    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"
Sep 07 10:27:39.235256 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.235176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x" event={"ID":"4d4c44fa-1851-45f7-a60b-3237c4ca9196","Type":"ContainerDied","Data":"d2b26bec567a4cc736d65ed8c70672f8566f2a79f35dbc039bab046d4b111e84"}
Sep 07 10:27:39.246065 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.246048    2560 scope.go:117] "RemoveContainer" containerID="a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f"
Sep 07 10:27:39.246406 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:27:39.246382    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f\": container with ID starting with a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f not found: ID does not exist" containerID="a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f"
Sep 07 10:27:39.246497 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.246415    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f"} err="failed to get container status \"a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f\": rpc error: code = NotFound desc = could not find container \"a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f\": container with ID starting with a9d7aa73dce2248ce9b266497e8e85f5f641476b5c91d818289ea32bbb15247f not found: ID does not exist"
Sep 07 10:27:39.246497 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.246432    2560 scope.go:117] "RemoveContainer" containerID="9e9ad46eb759114cac652b1b8c59004d473ec7b943e5425d902f176c4a810caa"
Sep 07 10:27:39.258971 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.258951    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"]
Sep 07 10:27:39.262669 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.262651    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-81af70-8684f866fc-nhd2p"]
Sep 07 10:27:39.273369 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.273348    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"]
Sep 07 10:27:39.277378 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.277345    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-81af70-85d4c48d6b-7tf4x"]
Sep 07 10:27:39.894592 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.894562    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4c44fa-1851-45f7-a60b-3237c4ca9196" path="/var/lib/kubelet/pods/4d4c44fa-1851-45f7-a60b-3237c4ca9196/volumes"
Sep 07 10:27:39.894910 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:39.894896    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="740ef2d2-baad-4e33-8335-ce68d166aa4b" path="/var/lib/kubelet/pods/740ef2d2-baad-4e33-8335-ce68d166aa4b/volumes"
Sep 07 10:27:44.489508 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:44.489468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:44.554833 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:44.554796    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:44.752909 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:44.752774    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:44.795253 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:44.795216    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:27:45.154897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:45.154863    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:45.462152 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:45.462029    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:45.616630 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:45.616585    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:47.182881 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:47.182851    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:47.281045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:47.281009    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:47.383196 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:47.383152    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:51.940262 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:51.940224    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:57.664760 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:57.664674    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:57.722581 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:57.722546    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:57.806447 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:57.806408    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:27:57.901229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:27:57.901190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:00.055083 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055036    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"]
Sep 07 10:28:00.055432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055377    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740ef2d2-baad-4e33-8335-ce68d166aa4b" containerName="payload-processing"
Sep 07 10:28:00.055432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055389    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="740ef2d2-baad-4e33-8335-ce68d166aa4b" containerName="payload-processing"
Sep 07 10:28:00.055432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055405    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4c44fa-1851-45f7-a60b-3237c4ca9196" containerName="maas-api"
Sep 07 10:28:00.055432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055410    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4c44fa-1851-45f7-a60b-3237c4ca9196" containerName="maas-api"
Sep 07 10:28:00.055432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055422    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff59943-d4b7-40c6-a71f-5ada380495a5" containerName="payload-pre-processing"
Sep 07 10:28:00.055432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055428    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff59943-d4b7-40c6-a71f-5ada380495a5" containerName="payload-pre-processing"
Sep 07 10:28:00.055627 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055487    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff59943-d4b7-40c6-a71f-5ada380495a5" containerName="payload-pre-processing"
Sep 07 10:28:00.055627 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055497    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="740ef2d2-baad-4e33-8335-ce68d166aa4b" containerName="payload-processing"
Sep 07 10:28:00.055627 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.055504    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4c44fa-1851-45f7-a60b-3237c4ca9196" containerName="maas-api"
Sep 07 10:28:00.059839 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.059822    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.065402 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.065380    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2d94a5\"/\"openshift-service-ca.crt\""
Sep 07 10:28:00.066148 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.066011    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2d94a5\"/\"default-dockercfg-ztn4n\""
Sep 07 10:28:00.066148 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.066055    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2d94a5\"/\"kube-root-ca.crt\""
Sep 07 10:28:00.066148 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.066059    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2d94a5\"/\"noten-model-2d94a5-kserve-self-signed-certs\""
Sep 07 10:28:00.079398 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.079379    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"]
Sep 07 10:28:00.101532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.101511    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:00.135222 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.135200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1ac5e0-967d-42ee-ad1d-a58725983792-tls-certs\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.135326 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.135234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-home\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.135326 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.135263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-tmp-dir\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.135442 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.135374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-dshm\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.135442 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.135412    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbx4p\" (UniqueName: \"kubernetes.io/projected/6c1ac5e0-967d-42ee-ad1d-a58725983792-kube-api-access-nbx4p\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.135541 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.135443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-model-cache\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236222 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1ac5e0-967d-42ee-ad1d-a58725983792-tls-certs\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236351 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-home\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236351 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-tmp-dir\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236351 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-dshm\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbx4p\" (UniqueName: \"kubernetes.io/projected/6c1ac5e0-967d-42ee-ad1d-a58725983792-kube-api-access-nbx4p\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-model-cache\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236654 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-home\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236720 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236641    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-tmp-dir\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.236771 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.236753    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-model-cache\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.238341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.238323    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-dshm\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.238632 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.238615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1ac5e0-967d-42ee-ad1d-a58725983792-tls-certs\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.245780 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.245758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbx4p\" (UniqueName: \"kubernetes.io/projected/6c1ac5e0-967d-42ee-ad1d-a58725983792-kube-api-access-nbx4p\") pod \"noten-model-2d94a5-kserve-868b6db679-wpsn9\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") " pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.368665 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.368611    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:00.643662 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:00.643637    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"]
Sep 07 10:28:00.645034 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:28:00.645010    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c1ac5e0_967d_42ee_ad1d_a58725983792.slice/crio-04757d4b1ac38eefe2c34a5caaf4918d0a059f6a7105d1dd859ed9f7e7394622 WatchSource:0}: Error finding container 04757d4b1ac38eefe2c34a5caaf4918d0a059f6a7105d1dd859ed9f7e7394622: Status 404 returned error can't find the container with id 04757d4b1ac38eefe2c34a5caaf4918d0a059f6a7105d1dd859ed9f7e7394622
Sep 07 10:28:01.342716 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:01.342687    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9" event={"ID":"6c1ac5e0-967d-42ee-ad1d-a58725983792","Type":"ContainerStarted","Data":"79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e"}
Sep 07 10:28:01.343112 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:01.342724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9" event={"ID":"6c1ac5e0-967d-42ee-ad1d-a58725983792","Type":"ContainerStarted","Data":"04757d4b1ac38eefe2c34a5caaf4918d0a059f6a7105d1dd859ed9f7e7394622"}
Sep 07 10:28:01.368170 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:01.368123    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9" podStartSLOduration=1.3681072429999999 podStartE2EDuration="1.368107243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:01.367399766 +0000 UTC m=+987.915914753" watchObservedRunningTime="2026-09-07 10:28:01.368107243 +0000 UTC m=+987.916622229"
Sep 07 10:28:08.351265 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:08.351234    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:09.084309 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:09.084270    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:10.369285 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:10.369251    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:10.369697 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:10.369393    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:10.382383 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:10.382358    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:10.393979 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:10.393958    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:10.592784 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:10.592744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:28:14.741657 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:14.741621    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"]
Sep 07 10:28:14.762038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:14.762002    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:15.058594 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:15.058556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:15.173816 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:15.173779    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:15.266038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:15.266004    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:15.359889 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:15.359801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:15.768430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:15.768348    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"]
Sep 07 10:28:15.788901 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:15.788868    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:15.917758 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:15.917725    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:16.015065 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:16.015035    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:16.117179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:16.117143    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:16.270361 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:16.270325    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:16.391360 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:16.391268    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:16.797778 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:16.797742    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:17.127654 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:17.127545    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:17.607960 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:17.607921    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:20.200444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:20.200413    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:28:21.272280 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:21.272251    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:23.314709 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:23.314675    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:23.414533 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:23.414503    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:23.945140 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:23.945114    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:27.481401 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:27.481360    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:27.551124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:27.551086    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:27.607593 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:27.607562    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:27.670431 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:27.670396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:27.727056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:27.727021    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:28.013779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:28.013740    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:29.760906 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:29.760864    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v" podUID="96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" containerName="main" containerID="cri-o://5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174" gracePeriod=45
Sep 07 10:28:29.830236 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:29.830207    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:29.883913 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:29.883885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:30.000851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.000826    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:28:30.036503 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.036473    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:30.091266 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-home\") pod \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") "
Sep 07 10:28:30.091379 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091270    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-dshm\") pod \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") "
Sep 07 10:28:30.091379 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l6tj\" (UniqueName: \"kubernetes.io/projected/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-kube-api-access-5l6tj\") pod \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") "
Sep 07 10:28:30.091379 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091334    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tls-certs\") pod \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") "
Sep 07 10:28:30.091502 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091384    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-model-cache\") pod \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") "
Sep 07 10:28:30.091502 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tmp-dir\") pod \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\" (UID: \"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc\") "
Sep 07 10:28:30.091577 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-home" (OuterVolumeSpecName: "home") pod "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" (UID: "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:28:30.091695 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091667    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-model-cache" (OuterVolumeSpecName: "model-cache") pod "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" (UID: "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:28:30.091798 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091773    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-model-cache\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:30.091798 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091790    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-home\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:30.091882 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.091852    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" (UID: "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:28:30.093542 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.093511    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-dshm" (OuterVolumeSpecName: "dshm") pod "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" (UID: "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:28:30.093625 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.093546    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" (UID: "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:28:30.093625 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.093579    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-kube-api-access-5l6tj" (OuterVolumeSpecName: "kube-api-access-5l6tj") pod "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" (UID: "96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc"). InnerVolumeSpecName "kube-api-access-5l6tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:28:30.193041 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.193010    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tmp-dir\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:30.193041 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.193035    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-dshm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:30.193041 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.193044    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l6tj\" (UniqueName: \"kubernetes.io/projected/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-kube-api-access-5l6tj\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:30.193279 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.193053    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc-tls-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:30.342052 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.341969    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:30.472857 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.472817    2560 generic.go:356] "Generic (PLEG): container finished" podID="96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" containerID="5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174" exitCode=0
Sep 07 10:28:30.472857 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.472855    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"
Sep 07 10:28:30.473090 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.472894    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v" event={"ID":"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc","Type":"ContainerDied","Data":"5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174"}
Sep 07 10:28:30.473090 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.472941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v" event={"ID":"96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc","Type":"ContainerDied","Data":"12d478b2dd90b594fc473d28d3bbf1b0412e0adc4d90a87314641d86ad80f25b"}
Sep 07 10:28:30.473090 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.472959    2560 scope.go:117] "RemoveContainer" containerID="5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174"
Sep 07 10:28:30.481747 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.481720    2560 scope.go:117] "RemoveContainer" containerID="5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174"
Sep 07 10:28:30.482007 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:28:30.481988    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174\": container with ID starting with 5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174 not found: ID does not exist" containerID="5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174"
Sep 07 10:28:30.482058 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.482017    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174"} err="failed to get container status \"5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174\": rpc error: code = NotFound desc = could not find container \"5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174\": container with ID starting with 5d38f15c0e2e02134078d3486deed13d0743be96e61e0a2df1d41bf812c91174 not found: ID does not exist"
Sep 07 10:28:30.493578 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.493557    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"]
Sep 07 10:28:30.495046 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.495028    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-05ecbf/test-model-05ecbf-kserve-7f7f5dd96c-25t2v"]
Sep 07 10:28:30.685275 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.685186    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:30.787994 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:30.787956    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9" podUID="6c1ac5e0-967d-42ee-ad1d-a58725983792" containerName="main" containerID="cri-o://79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e" gracePeriod=45
Sep 07 10:28:31.030383 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.030362    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:31.102779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.102752    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1ac5e0-967d-42ee-ad1d-a58725983792-tls-certs\") pod \"6c1ac5e0-967d-42ee-ad1d-a58725983792\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") "
Sep 07 10:28:31.102912 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.102788    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-tmp-dir\") pod \"6c1ac5e0-967d-42ee-ad1d-a58725983792\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") "
Sep 07 10:28:31.102912 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.102810    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbx4p\" (UniqueName: \"kubernetes.io/projected/6c1ac5e0-967d-42ee-ad1d-a58725983792-kube-api-access-nbx4p\") pod \"6c1ac5e0-967d-42ee-ad1d-a58725983792\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") "
Sep 07 10:28:31.102912 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.102845    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-home\") pod \"6c1ac5e0-967d-42ee-ad1d-a58725983792\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") "
Sep 07 10:28:31.103116 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.102921    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-dshm\") pod \"6c1ac5e0-967d-42ee-ad1d-a58725983792\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") "
Sep 07 10:28:31.103116 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.102949    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-model-cache\") pod \"6c1ac5e0-967d-42ee-ad1d-a58725983792\" (UID: \"6c1ac5e0-967d-42ee-ad1d-a58725983792\") "
Sep 07 10:28:31.103116 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.103014    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c1ac5e0-967d-42ee-ad1d-a58725983792" (UID: "6c1ac5e0-967d-42ee-ad1d-a58725983792"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:28:31.103244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.103153    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-home" (OuterVolumeSpecName: "home") pod "6c1ac5e0-967d-42ee-ad1d-a58725983792" (UID: "6c1ac5e0-967d-42ee-ad1d-a58725983792"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:28:31.103294 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.103245    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c1ac5e0-967d-42ee-ad1d-a58725983792" (UID: "6c1ac5e0-967d-42ee-ad1d-a58725983792"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:28:31.103344 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.103305    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-home\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:31.103344 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.103321    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-model-cache\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:31.103344 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.103333    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-tmp-dir\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:31.105050 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.105031    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-dshm" (OuterVolumeSpecName: "dshm") pod "6c1ac5e0-967d-42ee-ad1d-a58725983792" (UID: "6c1ac5e0-967d-42ee-ad1d-a58725983792"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:28:31.105346 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.105331    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1ac5e0-967d-42ee-ad1d-a58725983792-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c1ac5e0-967d-42ee-ad1d-a58725983792" (UID: "6c1ac5e0-967d-42ee-ad1d-a58725983792"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:28:31.105405 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.105365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1ac5e0-967d-42ee-ad1d-a58725983792-kube-api-access-nbx4p" (OuterVolumeSpecName: "kube-api-access-nbx4p") pod "6c1ac5e0-967d-42ee-ad1d-a58725983792" (UID: "6c1ac5e0-967d-42ee-ad1d-a58725983792"). InnerVolumeSpecName "kube-api-access-nbx4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:28:31.204474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.204415    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbx4p\" (UniqueName: \"kubernetes.io/projected/6c1ac5e0-967d-42ee-ad1d-a58725983792-kube-api-access-nbx4p\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:31.204474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.204437    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1ac5e0-967d-42ee-ad1d-a58725983792-dshm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:31.204474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.204449    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1ac5e0-967d-42ee-ad1d-a58725983792-tls-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:31.478811 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.478743    2560 generic.go:356] "Generic (PLEG): container finished" podID="6c1ac5e0-967d-42ee-ad1d-a58725983792" containerID="79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e" exitCode=0
Sep 07 10:28:31.478811 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.478785    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"
Sep 07 10:28:31.479018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.478826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9" event={"ID":"6c1ac5e0-967d-42ee-ad1d-a58725983792","Type":"ContainerDied","Data":"79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e"}
Sep 07 10:28:31.479018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.478860    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9" event={"ID":"6c1ac5e0-967d-42ee-ad1d-a58725983792","Type":"ContainerDied","Data":"04757d4b1ac38eefe2c34a5caaf4918d0a059f6a7105d1dd859ed9f7e7394622"}
Sep 07 10:28:31.479018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.478886    2560 scope.go:117] "RemoveContainer" containerID="79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e"
Sep 07 10:28:31.490746 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.490730    2560 scope.go:117] "RemoveContainer" containerID="79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e"
Sep 07 10:28:31.491038 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:28:31.491020    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e\": container with ID starting with 79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e not found: ID does not exist" containerID="79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e"
Sep 07 10:28:31.491122 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.491049    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e"} err="failed to get container status \"79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e\": rpc error: code = NotFound desc = could not find container \"79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e\": container with ID starting with 79b8b58056461343316d543896271cf1211abda2cdca9567d383cd67b217cd5e not found: ID does not exist"
Sep 07 10:28:31.501480 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.501464    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"]
Sep 07 10:28:31.502877 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.502860    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2d94a5/noten-model-2d94a5-kserve-868b6db679-wpsn9"]
Sep 07 10:28:31.895555 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.895532    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c1ac5e0-967d-42ee-ad1d-a58725983792" path="/var/lib/kubelet/pods/6c1ac5e0-967d-42ee-ad1d-a58725983792/volumes"
Sep 07 10:28:31.895898 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:31.895885    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" path="/var/lib/kubelet/pods/96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc/volumes"
Sep 07 10:28:32.897373 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:32.897340    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:28:33.098445 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:33.098409    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:33.193208 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:33.193127    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:34.205117 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:34.205019    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:34.305677 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:34.305636    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:34.406766 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:34.406737    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:34.505992 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:34.505900    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:34.601208 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:34.601164    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:34.671269 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:34.671240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:34.764599 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:34.764523    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:34.839683 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:34.839649    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:35.238717 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:35.238688    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:36.119229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:36.119196    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:44.196701 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:44.196666    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:28:45.047879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.047845    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:45.602961 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.602927    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"]
Sep 07 10:28:45.603361 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.603206    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x" podUID="8ec0645f-1c1a-4e1c-9d02-f954812e9a72" containerName="maas-api" containerID="cri-o://2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6" gracePeriod=30
Sep 07 10:28:45.629231 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.629201    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:45.653573 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.653549    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"]
Sep 07 10:28:45.653805 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.653781    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b" podUID="07757aab-510d-478f-be6e-ad0a4f64f1a3" containerName="payload-processing" containerID="cri-o://c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2" gracePeriod=30
Sep 07 10:28:45.728446 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.728417    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:45.818226 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.818184    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:45.879677 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.879647    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:45.884286 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.884262    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:28:45.914192 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:45.914171    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:28:46.033961 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.033935    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-openshift-service-ca\") pod \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") "
Sep 07 10:28:46.034124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.033995    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07757aab-510d-478f-be6e-ad0a4f64f1a3-plugins-config-volume\") pod \"07757aab-510d-478f-be6e-ad0a4f64f1a3\" (UID: \"07757aab-510d-478f-be6e-ad0a4f64f1a3\") "
Sep 07 10:28:46.034124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.034023    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5xjc\" (UniqueName: \"kubernetes.io/projected/07757aab-510d-478f-be6e-ad0a4f64f1a3-kube-api-access-w5xjc\") pod \"07757aab-510d-478f-be6e-ad0a4f64f1a3\" (UID: \"07757aab-510d-478f-be6e-ad0a4f64f1a3\") "
Sep 07 10:28:46.034124 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.034097    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4ck4\" (UniqueName: \"kubernetes.io/projected/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-kube-api-access-w4ck4\") pod \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") "
Sep 07 10:28:46.034276 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.034228    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-metrics-certs\") pod \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") "
Sep 07 10:28:46.034335 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.034304    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls\") pod \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\" (UID: \"8ec0645f-1c1a-4e1c-9d02-f954812e9a72\") "
Sep 07 10:28:46.034394 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.034340    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ec0645f-1c1a-4e1c-9d02-f954812e9a72" (UID: "8ec0645f-1c1a-4e1c-9d02-f954812e9a72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:28:46.034444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.034409    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07757aab-510d-478f-be6e-ad0a4f64f1a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07757aab-510d-478f-be6e-ad0a4f64f1a3" (UID: "07757aab-510d-478f-be6e-ad0a4f64f1a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:28:46.034625 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.034608    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07757aab-510d-478f-be6e-ad0a4f64f1a3-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:46.034708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.034626    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-openshift-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:46.036260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.036231    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8ec0645f-1c1a-4e1c-9d02-f954812e9a72" (UID: "8ec0645f-1c1a-4e1c-9d02-f954812e9a72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:28:46.036260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.036232    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-kube-api-access-w4ck4" (OuterVolumeSpecName: "kube-api-access-w4ck4") pod "8ec0645f-1c1a-4e1c-9d02-f954812e9a72" (UID: "8ec0645f-1c1a-4e1c-9d02-f954812e9a72"). InnerVolumeSpecName "kube-api-access-w4ck4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:28:46.036416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.036258    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07757aab-510d-478f-be6e-ad0a4f64f1a3-kube-api-access-w5xjc" (OuterVolumeSpecName: "kube-api-access-w5xjc") pod "07757aab-510d-478f-be6e-ad0a4f64f1a3" (UID: "07757aab-510d-478f-be6e-ad0a4f64f1a3"). InnerVolumeSpecName "kube-api-access-w5xjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:28:46.036470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.036423    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ec0645f-1c1a-4e1c-9d02-f954812e9a72" (UID: "8ec0645f-1c1a-4e1c-9d02-f954812e9a72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:28:46.135972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.135922    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-metrics-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:46.135972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.135941    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-maas-api-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:46.135972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.135951    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5xjc\" (UniqueName: \"kubernetes.io/projected/07757aab-510d-478f-be6e-ad0a4f64f1a3-kube-api-access-w5xjc\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:46.135972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.135961    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ck4\" (UniqueName: \"kubernetes.io/projected/8ec0645f-1c1a-4e1c-9d02-f954812e9a72-kube-api-access-w4ck4\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:28:46.169211 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.169182    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:46.555348 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.555317    2560 generic.go:356] "Generic (PLEG): container finished" podID="07757aab-510d-478f-be6e-ad0a4f64f1a3" containerID="c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2" exitCode=0
Sep 07 10:28:46.555494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.555362    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"
Sep 07 10:28:46.555494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.555402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b" event={"ID":"07757aab-510d-478f-be6e-ad0a4f64f1a3","Type":"ContainerDied","Data":"c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2"}
Sep 07 10:28:46.555494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.555453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b" event={"ID":"07757aab-510d-478f-be6e-ad0a4f64f1a3","Type":"ContainerDied","Data":"f0bf1462138ab1f1c1b3f0c107b5478e1440d075b5395e809575574242256e51"}
Sep 07 10:28:46.555494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.555474    2560 scope.go:117] "RemoveContainer" containerID="c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2"
Sep 07 10:28:46.556830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.556808    2560 generic.go:356] "Generic (PLEG): container finished" podID="8ec0645f-1c1a-4e1c-9d02-f954812e9a72" containerID="2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6" exitCode=0
Sep 07 10:28:46.556964 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.556846    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"
Sep 07 10:28:46.556964 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.556837    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x" event={"ID":"8ec0645f-1c1a-4e1c-9d02-f954812e9a72","Type":"ContainerDied","Data":"2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6"}
Sep 07 10:28:46.556964 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.556903    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x" event={"ID":"8ec0645f-1c1a-4e1c-9d02-f954812e9a72","Type":"ContainerDied","Data":"dc4ad62f924370a2ce8e3b2d4edae819811608bc7412fc8d525db91943b3091a"}
Sep 07 10:28:46.564701 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.564541    2560 scope.go:117] "RemoveContainer" containerID="c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2"
Sep 07 10:28:46.564922 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:28:46.564884    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2\": container with ID starting with c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2 not found: ID does not exist" containerID="c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2"
Sep 07 10:28:46.564982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.564932    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2"} err="failed to get container status \"c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2\": rpc error: code = NotFound desc = could not find container \"c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2\": container with ID starting with c627272b920b29d9feb610931967557f065fb9d746004cde9ee16a637c0f48e2 not found: ID does not exist"
Sep 07 10:28:46.564982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.564948    2560 scope.go:117] "RemoveContainer" containerID="2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6"
Sep 07 10:28:46.572773 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.572755    2560 scope.go:117] "RemoveContainer" containerID="2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6"
Sep 07 10:28:46.573017 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:28:46.573000    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6\": container with ID starting with 2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6 not found: ID does not exist" containerID="2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6"
Sep 07 10:28:46.573109 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.573027    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6"} err="failed to get container status \"2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6\": rpc error: code = NotFound desc = could not find container \"2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6\": container with ID starting with 2a5e2681a8742d890424708b0eda1ea13bbef69d846d40a7119ab2cfbbd514a6 not found: ID does not exist"
Sep 07 10:28:46.579911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.579890    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"]
Sep 07 10:28:46.583326 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.583307    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f17c1f-587fb546f9-bdd6b"]
Sep 07 10:28:46.593798 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.593777    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"]
Sep 07 10:28:46.596969 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:46.596949    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f17c1f-598f5bb48c-gsq8x"]
Sep 07 10:28:47.895601 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:47.895571    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07757aab-510d-478f-be6e-ad0a4f64f1a3" path="/var/lib/kubelet/pods/07757aab-510d-478f-be6e-ad0a4f64f1a3/volumes"
Sep 07 10:28:47.895963 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:47.895948    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec0645f-1c1a-4e1c-9d02-f954812e9a72" path="/var/lib/kubelet/pods/8ec0645f-1c1a-4e1c-9d02-f954812e9a72/volumes"
Sep 07 10:28:52.383212 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:52.383161    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:52.450542 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:52.450516    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:52.632187 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:52.632131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:52.995848 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:52.995817    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:28:53.181345 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:53.181306    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:53.485218 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:53.485190    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:53.617514 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:53.617479    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:55.224298 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:55.224263    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:56.249552 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:56.249521    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:56.342509 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:56.342468    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:57.433312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.433282    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:57.449667 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.449640    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"]
Sep 07 10:28:57.449964 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.449952    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c1ac5e0-967d-42ee-ad1d-a58725983792" containerName="main"
Sep 07 10:28:57.450006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.449965    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c1ac5e0-967d-42ee-ad1d-a58725983792" containerName="main"
Sep 07 10:28:57.450006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.449975    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" containerName="main"
Sep 07 10:28:57.450006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.449980    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" containerName="main"
Sep 07 10:28:57.450006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.449993    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec0645f-1c1a-4e1c-9d02-f954812e9a72" containerName="maas-api"
Sep 07 10:28:57.450006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.449999    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec0645f-1c1a-4e1c-9d02-f954812e9a72" containerName="maas-api"
Sep 07 10:28:57.450180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.450011    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07757aab-510d-478f-be6e-ad0a4f64f1a3" containerName="payload-processing"
Sep 07 10:28:57.450180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.450016    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="07757aab-510d-478f-be6e-ad0a4f64f1a3" containerName="payload-processing"
Sep 07 10:28:57.450180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.450064    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec0645f-1c1a-4e1c-9d02-f954812e9a72" containerName="maas-api"
Sep 07 10:28:57.450180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.450085    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="07757aab-510d-478f-be6e-ad0a4f64f1a3" containerName="payload-processing"
Sep 07 10:28:57.450180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.450092    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c1ac5e0-967d-42ee-ad1d-a58725983792" containerName="main"
Sep 07 10:28:57.450180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.450099    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d3ec45-7be2-4240-bdbd-8b1fc5cfcdfc" containerName="main"
Sep 07 10:28:57.454567 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.454553    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.456844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.456823    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-0a577c-serving-cert\""
Sep 07 10:28:57.456964 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.456872    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 10:28:57.457739 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.457721    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 10:28:57.457821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.457742    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2ldcm\""
Sep 07 10:28:57.457821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.457757    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 10:28:57.463257 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.463237    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"]
Sep 07 10:28:57.527086 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.527046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96nz\" (UniqueName: \"kubernetes.io/projected/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-kube-api-access-f96nz\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.527192 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.527139    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.527192 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.527182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.527280 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.527210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.534723 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.534696    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:57.628485 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.628462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.628585 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.628514    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96nz\" (UniqueName: \"kubernetes.io/projected/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-kube-api-access-f96nz\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.628585 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.628554    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.628676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.628584    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.628676 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:28:57.628668    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-serving-cert: secret "maas-api-e2e-shared-a-w3-0a577c-serving-cert" not found
Sep 07 10:28:57.628748 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:28:57.628726    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls podName:768ed424-fdfc-4da3-ae52-e7d7a0e1812d nodeName:}" failed. No retries permitted until 2026-09-07 10:28:58.128710514 +0000 UTC m=+1044.677225478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6" (UID: "768ed424-fdfc-4da3-ae52-e7d7a0e1812d") : secret "maas-api-e2e-shared-a-w3-0a577c-serving-cert" not found
Sep 07 10:28:57.629204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.629179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.630817 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.630801    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.642178 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.642156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96nz\" (UniqueName: \"kubernetes.io/projected/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-kube-api-access-f96nz\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:57.902590 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:57.902557    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:58.004436 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.004398    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:58.022225 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.022198    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"]
Sep 07 10:28:58.025586 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.025570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:58.028242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.028218    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0a577c-dockercfg-ssl48\""
Sep 07 10:28:58.028355 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.028254    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0a577c\""
Sep 07 10:28:58.042036 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.042013    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"]
Sep 07 10:28:58.133082 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.133042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1470fc6-7b1c-4579-8195-8e64b1a72157-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c\" (UID: \"d1470fc6-7b1c-4579-8195-8e64b1a72157\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:58.133198 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.133119    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzsk6\" (UniqueName: \"kubernetes.io/projected/d1470fc6-7b1c-4579-8195-8e64b1a72157-kube-api-access-zzsk6\") pod \"payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c\" (UID: \"d1470fc6-7b1c-4579-8195-8e64b1a72157\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:58.133198 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.133154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:58.135326 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.135306    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:58.234547 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.234472    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1470fc6-7b1c-4579-8195-8e64b1a72157-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c\" (UID: \"d1470fc6-7b1c-4579-8195-8e64b1a72157\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:58.234547 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.234538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzsk6\" (UniqueName: \"kubernetes.io/projected/d1470fc6-7b1c-4579-8195-8e64b1a72157-kube-api-access-zzsk6\") pod \"payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c\" (UID: \"d1470fc6-7b1c-4579-8195-8e64b1a72157\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:58.235188 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.235171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1470fc6-7b1c-4579-8195-8e64b1a72157-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c\" (UID: \"d1470fc6-7b1c-4579-8195-8e64b1a72157\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:58.242570 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.242549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzsk6\" (UniqueName: \"kubernetes.io/projected/d1470fc6-7b1c-4579-8195-8e64b1a72157-kube-api-access-zzsk6\") pod \"payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c\" (UID: \"d1470fc6-7b1c-4579-8195-8e64b1a72157\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:58.336015 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.335986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:58.365657 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.365634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:58.565263 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.565233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:58.605668 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.605571    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"]
Sep 07 10:28:58.608229 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:28:58.608193    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1470fc6_7b1c_4579_8195_8e64b1a72157.slice/crio-f61c40f6bf2cc2e59715e379c44288265b77adb8313c145929d0009e5289a4c5 WatchSource:0}: Error finding container f61c40f6bf2cc2e59715e379c44288265b77adb8313c145929d0009e5289a4c5: Status 404 returned error can't find the container with id f61c40f6bf2cc2e59715e379c44288265b77adb8313c145929d0009e5289a4c5
Sep 07 10:28:58.620863 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.620836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c" event={"ID":"d1470fc6-7b1c-4579-8195-8e64b1a72157","Type":"ContainerStarted","Data":"f61c40f6bf2cc2e59715e379c44288265b77adb8313c145929d0009e5289a4c5"}
Sep 07 10:28:58.645449 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.645429    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"]
Sep 07 10:28:58.647620 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:28:58.647598    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768ed424_fdfc_4da3_ae52_e7d7a0e1812d.slice/crio-e1ff715138fd735428995ea1f2df6a0be0fb36b214f9948d4ac95a9b092e192f WatchSource:0}: Error finding container e1ff715138fd735428995ea1f2df6a0be0fb36b214f9948d4ac95a9b092e192f: Status 404 returned error can't find the container with id e1ff715138fd735428995ea1f2df6a0be0fb36b214f9948d4ac95a9b092e192f
Sep 07 10:28:58.649315 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.649298    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:28:58.650231 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.650202    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:58.962339 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:58.962258    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:59.384816 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:59.384781    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:28:59.629230 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:59.629183    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6" event={"ID":"768ed424-fdfc-4da3-ae52-e7d7a0e1812d","Type":"ContainerStarted","Data":"b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b"}
Sep 07 10:28:59.629637 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:59.629251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6" event={"ID":"768ed424-fdfc-4da3-ae52-e7d7a0e1812d","Type":"ContainerStarted","Data":"e1ff715138fd735428995ea1f2df6a0be0fb36b214f9948d4ac95a9b092e192f"}
Sep 07 10:28:59.629637 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:59.629281    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:28:59.630580 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:59.630555    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c" event={"ID":"d1470fc6-7b1c-4579-8195-8e64b1a72157","Type":"ContainerStarted","Data":"99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa"}
Sep 07 10:28:59.630687 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:59.630620    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:28:59.649890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:59.649816    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6" podStartSLOduration=2.251032788 podStartE2EDuration="2.649801284s" totalImagesPullingTime="398.768496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:59.648863776 +0000 UTC m=+1046.197378765" watchObservedRunningTime="2026-09-07 10:28:59.649801284 +0000 UTC m=+1046.198316270"
Sep 07 10:28:59.673007 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:28:59.672964    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c" podStartSLOduration=1.672950513 podStartE2EDuration="1.672950513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:28:59.672239225 +0000 UTC m=+1046.220754215" watchObservedRunningTime="2026-09-07 10:28:59.672950513 +0000 UTC m=+1046.221465499"
Sep 07 10:29:00.494011 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:00.493972    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:29:04.677923 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:04.677888    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:04.779097 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:04.779040    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:05.107623 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:05.107591    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:05.640961 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:05.640932    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:29:10.637612 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:10.637584    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:29:15.471662 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:15.471621    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:15.561666 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:15.561634    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:15.651351 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:15.651311    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:15.712849 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:15.712818    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:16.019395 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:16.019364    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:18.189384 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:18.189357    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:29:18.301482 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:18.301451    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:21.147528 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:21.147492    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:21.455520 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:21.455429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:21.773989 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:21.773913    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:22.106314 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:22.106278    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:22.294026 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:22.293998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:22.395147 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:22.395065    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:22.574083 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:22.574047    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:22.934155 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:22.934122    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:23.059907 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:23.059874    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:23.128033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:23.127999    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:24.453108 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:24.453059    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:24.549384 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:24.549346    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:25.734941 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:25.734907    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:25.789437 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:25.789401    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:26.228471 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.228441    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:26.314922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.314846    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:26.346688 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.346664    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"]
Sep 07 10:29:26.350285 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.350269    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:26.356569 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.356546    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-93e41e\""
Sep 07 10:29:26.357053 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.356566    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-93e41e-dockercfg-m9wwn\""
Sep 07 10:29:26.365275 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.365251    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"]
Sep 07 10:29:26.460705 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.460674    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q\" (UID: \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:26.460843 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.460728    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbp9\" (UniqueName: \"kubernetes.io/projected/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-kube-api-access-kmbp9\") pod \"payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q\" (UID: \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:26.561490 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.561461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbp9\" (UniqueName: \"kubernetes.io/projected/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-kube-api-access-kmbp9\") pod \"payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q\" (UID: \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:26.561605 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.561551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q\" (UID: \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:26.562284 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.562266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q\" (UID: \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:26.570534 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.570506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbp9\" (UniqueName: \"kubernetes.io/projected/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-kube-api-access-kmbp9\") pod \"payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q\" (UID: \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:26.660056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.660014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:26.934047 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:26.934022    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"]
Sep 07 10:29:26.936317 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:29:26.936282    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7f9cec_a5ad_4df1_97c4_3d5bcd8cef6d.slice/crio-7114074a2ea01e82a270998a62d412ca10f65e08768fb6a802a2c70a3c7e1b33 WatchSource:0}: Error finding container 7114074a2ea01e82a270998a62d412ca10f65e08768fb6a802a2c70a3c7e1b33: Status 404 returned error can't find the container with id 7114074a2ea01e82a270998a62d412ca10f65e08768fb6a802a2c70a3c7e1b33
Sep 07 10:29:27.754467 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:27.754426    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q" event={"ID":"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d","Type":"ContainerStarted","Data":"88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81"}
Sep 07 10:29:27.754632 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:27.754487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q" event={"ID":"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d","Type":"ContainerStarted","Data":"7114074a2ea01e82a270998a62d412ca10f65e08768fb6a802a2c70a3c7e1b33"}
Sep 07 10:29:27.754632 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:27.754567    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:27.782558 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:27.782519    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q" podStartSLOduration=1.7825069249999999 podStartE2EDuration="1.782506925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:27.782232981 +0000 UTC m=+1074.330747968" watchObservedRunningTime="2026-09-07 10:29:27.782506925 +0000 UTC m=+1074.331021950"
Sep 07 10:29:28.868171 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:28.868138    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:28.969852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:28.969820    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:29.068376 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:29.068351    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:29.203934 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:29.203850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:29.399624 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:29.399584    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:29:34.487900 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:34.487858    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:34.587927 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:34.587894    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:34.688829 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:34.688795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:34.905842 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:34.905814    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:34.957466 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:34.957438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:35.817681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:35.817649    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:36.102660 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:36.102576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:36.209542 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:36.209516    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:38.761437 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:38.761409    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:29:39.519492 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:39.519461    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:39.619617 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:39.619584    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:40.675318 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:40.675280    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:40.775979 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:40.775948    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:40.967092 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:40.967003    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:41.281343 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.281253    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:41.381807 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.381771    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:41.483817 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.483780    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"]
Sep 07 10:29:41.488784 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.488763    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:41.491435 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.491415    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7bdcc7\""
Sep 07 10:29:41.491529 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.491439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7bdcc7-dockercfg-mkrfw\""
Sep 07 10:29:41.500665 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.500641    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"]
Sep 07 10:29:41.583055 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.583025    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvgxp\" (UniqueName: \"kubernetes.io/projected/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-kube-api-access-rvgxp\") pod \"payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz\" (UID: \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:41.583241 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.583078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz\" (UID: \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:41.684494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.684463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvgxp\" (UniqueName: \"kubernetes.io/projected/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-kube-api-access-rvgxp\") pod \"payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz\" (UID: \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:41.684833 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.684505    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz\" (UID: \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:41.685229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.685209    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz\" (UID: \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:41.692900 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.692877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvgxp\" (UniqueName: \"kubernetes.io/projected/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-kube-api-access-rvgxp\") pod \"payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz\" (UID: \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:41.798511 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:41.798479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:42.077192 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:42.077167    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"]
Sep 07 10:29:42.079127 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:29:42.079102    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ff1335_e8ff_460e_ae49_cb2f3b160df1.slice/crio-718072911f7412bf77e9e7c24f549315e1547f288709b9c7213db23da2b55d93 WatchSource:0}: Error finding container 718072911f7412bf77e9e7c24f549315e1547f288709b9c7213db23da2b55d93: Status 404 returned error can't find the container with id 718072911f7412bf77e9e7c24f549315e1547f288709b9c7213db23da2b55d93
Sep 07 10:29:42.823678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:42.823650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz" event={"ID":"e6ff1335-e8ff-460e-ae49-cb2f3b160df1","Type":"ContainerStarted","Data":"afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744"}
Sep 07 10:29:42.824050 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:42.823698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz" event={"ID":"e6ff1335-e8ff-460e-ae49-cb2f3b160df1","Type":"ContainerStarted","Data":"718072911f7412bf77e9e7c24f549315e1547f288709b9c7213db23da2b55d93"}
Sep 07 10:29:42.824050 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:42.823719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:42.843242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:42.843199    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz" podStartSLOduration=1.8431827520000001 podStartE2EDuration="1.843182752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:42.840928885 +0000 UTC m=+1089.389443874" watchObservedRunningTime="2026-09-07 10:29:42.843182752 +0000 UTC m=+1089.391697739"
Sep 07 10:29:44.895343 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:44.895299    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:29:46.582492 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:46.582462    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:49.366130 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:49.366100    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:53.830144 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:53.830111    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:29:56.793005 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.792975    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"]
Sep 07 10:29:56.798492 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.798462    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:56.803121 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.803087    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"openshift-service-ca.crt\""
Sep 07 10:29:56.803121 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.803109    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"default-dockercfg-c5g9z\""
Sep 07 10:29:56.803334 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.803222    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"kube-root-ca.crt\""
Sep 07 10:29:56.803416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.803359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"auth-test-model-0a577c-kserve-self-signed-certs\""
Sep 07 10:29:56.809819 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.809794    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"]
Sep 07 10:29:56.887147 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.887105    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:29:56.906747 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.906719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-home\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:56.906747 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.906748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2r2\" (UniqueName: \"kubernetes.io/projected/5fe04cb0-dc93-472d-9565-63ed960abffb-kube-api-access-jl2r2\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:56.906934 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.906769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe04cb0-dc93-472d-9565-63ed960abffb-tls-certs\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:56.906934 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.906838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-model-cache\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:56.906934 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.906865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-tmp-dir\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:56.906934 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:56.906906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-dshm\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.007780 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.007731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-home\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.007939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.007784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2r2\" (UniqueName: \"kubernetes.io/projected/5fe04cb0-dc93-472d-9565-63ed960abffb-kube-api-access-jl2r2\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.007939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.007802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe04cb0-dc93-472d-9565-63ed960abffb-tls-certs\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.007939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.007840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-model-cache\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.007939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.007862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-tmp-dir\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.008197 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.007985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-dshm\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.008197 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.008178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-home\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.008270 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.008199    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-model-cache\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.008270 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.008253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-tmp-dir\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.010094 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.010051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-dshm\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.010223 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.010205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe04cb0-dc93-472d-9565-63ed960abffb-tls-certs\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.015808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.015789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2r2\" (UniqueName: \"kubernetes.io/projected/5fe04cb0-dc93-472d-9565-63ed960abffb-kube-api-access-jl2r2\") pod \"auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.109916 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.109842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:29:57.381789 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.381763    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"]
Sep 07 10:29:57.383522 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:29:57.383495    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fe04cb0_dc93_472d_9565_63ed960abffb.slice/crio-dd1906ca49d529628eb896260db2eab4a525784fc17b4fe927728b13d642d43b WatchSource:0}: Error finding container dd1906ca49d529628eb896260db2eab4a525784fc17b4fe927728b13d642d43b: Status 404 returned error can't find the container with id dd1906ca49d529628eb896260db2eab4a525784fc17b4fe927728b13d642d43b
Sep 07 10:29:57.894995 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.894962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8" event={"ID":"5fe04cb0-dc93-472d-9565-63ed960abffb","Type":"ContainerStarted","Data":"94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5"}
Sep 07 10:29:57.895374 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.895007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8" event={"ID":"5fe04cb0-dc93-472d-9565-63ed960abffb","Type":"ContainerStarted","Data":"dd1906ca49d529628eb896260db2eab4a525784fc17b4fe927728b13d642d43b"}
Sep 07 10:29:57.908040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:57.907994    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8" podStartSLOduration=1.9079794730000001 podStartE2EDuration="1.907979473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:29:57.906588244 +0000 UTC m=+1104.455103233" watchObservedRunningTime="2026-09-07 10:29:57.907979473 +0000 UTC m=+1104.456494459"
Sep 07 10:29:59.490593 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:29:59.490560    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:30:00.139596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:00.139561    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx"]
Sep 07 10:30:00.143204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:00.143188    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx"
Sep 07 10:30:00.145759 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:00.145743    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\""
Sep 07 10:30:00.151812 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:00.151790    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx"]
Sep 07 10:30:00.155415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:00.155392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx"
Sep 07 10:30:00.435031 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:00.435005    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx"]
Sep 07 10:30:00.436598 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:30:00.436564    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c698d18_6ed4_4269_807a_8c503fbfbde2.slice/crio-4a615708393c90d29b6a48ae01a82c1dd8061f401e9b45c64a155f0b188e0caa WatchSource:0}: Error finding container 4a615708393c90d29b6a48ae01a82c1dd8061f401e9b45c64a155f0b188e0caa: Status 404 returned error can't find the container with id 4a615708393c90d29b6a48ae01a82c1dd8061f401e9b45c64a155f0b188e0caa
Sep 07 10:30:00.906752 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:00.906716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx" event={"ID":"1c698d18-6ed4-4269-807a-8c503fbfbde2","Type":"ContainerStarted","Data":"4a615708393c90d29b6a48ae01a82c1dd8061f401e9b45c64a155f0b188e0caa"}
Sep 07 10:30:02.919299 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:02.919262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx" event={"ID":"1c698d18-6ed4-4269-807a-8c503fbfbde2","Type":"ContainerStarted","Data":"d24d05dc23fccf9eaf70270e8516b41aa85782a040e662f4a78a0c9cf6883b58"}
Sep 07 10:30:02.937163 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:02.937052    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx" podStartSLOduration=0.945261728 podStartE2EDuration="2.937037682s" totalImagesPullingTime="1.991775954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:02.935421109 +0000 UTC m=+1109.483936096" watchObservedRunningTime="2026-09-07 10:30:02.937037682 +0000 UTC m=+1109.485552668"
Sep 07 10:30:03.038477 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:03.038443    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:03.924376 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:03.924340    2560 generic.go:356] "Generic (PLEG): container finished" podID="1c698d18-6ed4-4269-807a-8c503fbfbde2" containerID="d24d05dc23fccf9eaf70270e8516b41aa85782a040e662f4a78a0c9cf6883b58" exitCode=0
Sep 07 10:30:03.924767 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:03.924437    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx" event={"ID":"1c698d18-6ed4-4269-807a-8c503fbfbde2","Type":"ContainerDied","Data":"d24d05dc23fccf9eaf70270e8516b41aa85782a040e662f4a78a0c9cf6883b58"}
Sep 07 10:30:04.473554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:04.473518    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:04.791431 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:04.791396    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:05.063761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:05.063738    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx"
Sep 07 10:30:05.083990 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:05.083909    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:05.488254 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:05.488179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:05.935596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:05.935569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx" event={"ID":"1c698d18-6ed4-4269-807a-8c503fbfbde2","Type":"ContainerDied","Data":"4a615708393c90d29b6a48ae01a82c1dd8061f401e9b45c64a155f0b188e0caa"}
Sep 07 10:30:05.935596 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:05.935586    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812950-tmjhx"
Sep 07 10:30:05.935771 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:05.935612    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a615708393c90d29b6a48ae01a82c1dd8061f401e9b45c64a155f0b188e0caa"
Sep 07 10:30:07.110753 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:07.110718    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:30:07.111234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:07.110773    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:30:07.123286 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:07.123268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:30:07.732660 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:07.732628    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:07.837217 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:07.837186    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:07.956231 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:07.956201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:30:08.844134 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.844101    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"]
Sep 07 10:30:08.844530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.844453    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c698d18-6ed4-4269-807a-8c503fbfbde2" containerName="cleanup"
Sep 07 10:30:08.844530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.844464    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c698d18-6ed4-4269-807a-8c503fbfbde2" containerName="cleanup"
Sep 07 10:30:08.844655 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.844541    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c698d18-6ed4-4269-807a-8c503fbfbde2" containerName="cleanup"
Sep 07 10:30:08.847395 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.847376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:08.849790 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.849771    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-319d4c-serving-cert\""
Sep 07 10:30:08.856483 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.856454    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"]
Sep 07 10:30:08.870210 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.870159    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:08.910447 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.910419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:08.910548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.910455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:08.910548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.910504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkshk\" (UniqueName: \"kubernetes.io/projected/a4323079-7fdb-4156-92cb-61af62f109dd-kube-api-access-rkshk\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:08.910548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.910523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4323079-7fdb-4156-92cb-61af62f109dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:08.934535 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:08.934515    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:09.010873 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.010852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkshk\" (UniqueName: \"kubernetes.io/projected/a4323079-7fdb-4156-92cb-61af62f109dd-kube-api-access-rkshk\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.010972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.010879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4323079-7fdb-4156-92cb-61af62f109dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.010972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.010922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.010972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.010958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.011176 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:30:09.011048    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-serving-cert: secret "maas-api-e2e-shared-b-w2-319d4c-serving-cert" not found
Sep 07 10:30:09.011176 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:30:09.011146    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls podName:a4323079-7fdb-4156-92cb-61af62f109dd nodeName:}" failed. No retries permitted until 2026-09-07 10:30:09.511124549 +0000 UTC m=+1116.059639517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls") pod "maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q" (UID: "a4323079-7fdb-4156-92cb-61af62f109dd") : secret "maas-api-e2e-shared-b-w2-319d4c-serving-cert" not found
Sep 07 10:30:09.011581 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.011558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4323079-7fdb-4156-92cb-61af62f109dd-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.013401 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.013384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.021711 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.021689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkshk\" (UniqueName: \"kubernetes.io/projected/a4323079-7fdb-4156-92cb-61af62f109dd-kube-api-access-rkshk\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.369411 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.369378    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:09.469695 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.469668    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:09.515263 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.515234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.517517 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.517499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.522676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.522651    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"]
Sep 07 10:30:09.526855 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.526837    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:09.529414 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.529394    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-319d4c\""
Sep 07 10:30:09.529548 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.529533    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-319d4c-dockercfg-8bc5l\""
Sep 07 10:30:09.537494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.537473    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"]
Sep 07 10:30:09.616338 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.616310    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldrf\" (UniqueName: \"kubernetes.io/projected/281309a8-e027-425b-b375-4179545fc516-kube-api-access-2ldrf\") pod \"payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s\" (UID: \"281309a8-e027-425b-b375-4179545fc516\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:09.616456 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.616419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281309a8-e027-425b-b375-4179545fc516-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s\" (UID: \"281309a8-e027-425b-b375-4179545fc516\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:09.717144 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.717059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldrf\" (UniqueName: \"kubernetes.io/projected/281309a8-e027-425b-b375-4179545fc516-kube-api-access-2ldrf\") pod \"payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s\" (UID: \"281309a8-e027-425b-b375-4179545fc516\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:09.717244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.717164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281309a8-e027-425b-b375-4179545fc516-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s\" (UID: \"281309a8-e027-425b-b375-4179545fc516\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:09.717777 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.717759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281309a8-e027-425b-b375-4179545fc516-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s\" (UID: \"281309a8-e027-425b-b375-4179545fc516\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:09.728846 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.728820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldrf\" (UniqueName: \"kubernetes.io/projected/281309a8-e027-425b-b375-4179545fc516-kube-api-access-2ldrf\") pod \"payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s\" (UID: \"281309a8-e027-425b-b375-4179545fc516\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:09.758842 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.758823    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:09.837563 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:09.837538    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:10.026786 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.026764    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"]
Sep 07 10:30:10.028638 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:30:10.028613    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4323079_7fdb_4156_92cb_61af62f109dd.slice/crio-6f0b92f6581dd25a67b5df5ca57d894bda4f1deaaaf10b0db2f6e931604d7029 WatchSource:0}: Error finding container 6f0b92f6581dd25a67b5df5ca57d894bda4f1deaaaf10b0db2f6e931604d7029: Status 404 returned error can't find the container with id 6f0b92f6581dd25a67b5df5ca57d894bda4f1deaaaf10b0db2f6e931604d7029
Sep 07 10:30:10.107539 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.107519    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"]
Sep 07 10:30:10.109421 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:30:10.109399    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281309a8_e027_425b_b375_4179545fc516.slice/crio-764d99b94c7354303b195c149be45a293e04f615806686502fc65693b30758f9 WatchSource:0}: Error finding container 764d99b94c7354303b195c149be45a293e04f615806686502fc65693b30758f9: Status 404 returned error can't find the container with id 764d99b94c7354303b195c149be45a293e04f615806686502fc65693b30758f9
Sep 07 10:30:10.966937 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.966899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q" event={"ID":"a4323079-7fdb-4156-92cb-61af62f109dd","Type":"ContainerStarted","Data":"deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81"}
Sep 07 10:30:10.967146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.967087    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:10.967146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.967120    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q" event={"ID":"a4323079-7fdb-4156-92cb-61af62f109dd","Type":"ContainerStarted","Data":"6f0b92f6581dd25a67b5df5ca57d894bda4f1deaaaf10b0db2f6e931604d7029"}
Sep 07 10:30:10.968522 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.968489    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s" event={"ID":"281309a8-e027-425b-b375-4179545fc516","Type":"ContainerStarted","Data":"81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3"}
Sep 07 10:30:10.968652 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.968541    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:10.968652 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.968564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s" event={"ID":"281309a8-e027-425b-b375-4179545fc516","Type":"ContainerStarted","Data":"764d99b94c7354303b195c149be45a293e04f615806686502fc65693b30758f9"}
Sep 07 10:30:10.985395 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.985349    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q" podStartSLOduration=2.62897316 podStartE2EDuration="2.985336273s" totalImagesPullingTime="356.363113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:10.982949373 +0000 UTC m=+1117.531464360" watchObservedRunningTime="2026-09-07 10:30:10.985336273 +0000 UTC m=+1117.533851322"
Sep 07 10:30:10.999586 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:10.999541    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s" podStartSLOduration=1.999524606 podStartE2EDuration="1.999524606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:10.997835749 +0000 UTC m=+1117.546350736" watchObservedRunningTime="2026-09-07 10:30:10.999524606 +0000 UTC m=+1117.548039598"
Sep 07 10:30:16.096763 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:16.096731    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:30:16.978415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:16.978385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:30:21.976221 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:21.976190    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:30:32.682736 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:32.682689    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:33.938495 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:33.938467    2560 scope.go:117] "RemoveContainer" containerID="9d9909fc8bc1cebf2a1551682411a472104b01a291c2b2fa352676470ae91962"
Sep 07 10:30:40.247311 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.247283    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:40.249792 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.249776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:40.252118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.252096    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qpjjb\""
Sep 07 10:30:40.257457 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.257438    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:40.375544 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.375518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5pf\" (UniqueName: \"kubernetes.io/projected/95dd921b-958c-4fe1-860f-1bd7dbc33578-kube-api-access-xs5pf\") pod \"test-curl-8014-140224719722656\" (UID: \"95dd921b-958c-4fe1-860f-1bd7dbc33578\") " pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:40.475961 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.475931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5pf\" (UniqueName: \"kubernetes.io/projected/95dd921b-958c-4fe1-860f-1bd7dbc33578-kube-api-access-xs5pf\") pod \"test-curl-8014-140224719722656\" (UID: \"95dd921b-958c-4fe1-860f-1bd7dbc33578\") " pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:40.485595 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.485569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5pf\" (UniqueName: \"kubernetes.io/projected/95dd921b-958c-4fe1-860f-1bd7dbc33578-kube-api-access-xs5pf\") pod \"test-curl-8014-140224719722656\" (UID: \"95dd921b-958c-4fe1-860f-1bd7dbc33578\") " pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:40.560199 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.560173    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:40.827867 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:40.827792    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:40.830790 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:30:40.830761    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95dd921b_958c_4fe1_860f_1bd7dbc33578.slice/crio-db78070e96b5e51fec548441bc9f324362bb4f7da5a5458757659e9fd2ed5c45 WatchSource:0}: Error finding container db78070e96b5e51fec548441bc9f324362bb4f7da5a5458757659e9fd2ed5c45: Status 404 returned error can't find the container with id db78070e96b5e51fec548441bc9f324362bb4f7da5a5458757659e9fd2ed5c45
Sep 07 10:30:41.107813 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:41.107738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-140224719722656" event={"ID":"95dd921b-958c-4fe1-860f-1bd7dbc33578","Type":"ContainerStarted","Data":"db78070e96b5e51fec548441bc9f324362bb4f7da5a5458757659e9fd2ed5c45"}
Sep 07 10:30:42.113354 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:42.113314    2560 generic.go:356] "Generic (PLEG): container finished" podID="95dd921b-958c-4fe1-860f-1bd7dbc33578" containerID="92ee7defbd0aa16fa9a019c9755fd1f0de62f4484fd5af804c7e31147ac8d53e" exitCode=0
Sep 07 10:30:42.113731 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:42.113399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8014-140224719722656" event={"ID":"95dd921b-958c-4fe1-860f-1bd7dbc33578","Type":"ContainerDied","Data":"92ee7defbd0aa16fa9a019c9755fd1f0de62f4484fd5af804c7e31147ac8d53e"}
Sep 07 10:30:43.247479 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:43.247457    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:43.268505 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:43.268478    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-8014-140224719722656_95dd921b-958c-4fe1-860f-1bd7dbc33578/test-curl-8014-140224719722656/0.log"
Sep 07 10:30:43.294533 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:43.294506    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:43.298648 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:43.298620    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8014-140224719722656"]
Sep 07 10:30:43.400502 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:43.400412    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs5pf\" (UniqueName: \"kubernetes.io/projected/95dd921b-958c-4fe1-860f-1bd7dbc33578-kube-api-access-xs5pf\") pod \"95dd921b-958c-4fe1-860f-1bd7dbc33578\" (UID: \"95dd921b-958c-4fe1-860f-1bd7dbc33578\") "
Sep 07 10:30:43.402425 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:43.402399    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95dd921b-958c-4fe1-860f-1bd7dbc33578-kube-api-access-xs5pf" (OuterVolumeSpecName: "kube-api-access-xs5pf") pod "95dd921b-958c-4fe1-860f-1bd7dbc33578" (UID: "95dd921b-958c-4fe1-860f-1bd7dbc33578"). InnerVolumeSpecName "kube-api-access-xs5pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:30:43.501900 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:43.501872    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5pf\" (UniqueName: \"kubernetes.io/projected/95dd921b-958c-4fe1-860f-1bd7dbc33578-kube-api-access-xs5pf\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:30:43.896436 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:43.896395    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95dd921b-958c-4fe1-860f-1bd7dbc33578" path="/var/lib/kubelet/pods/95dd921b-958c-4fe1-860f-1bd7dbc33578/volumes"
Sep 07 10:30:44.124004 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:44.123977    2560 scope.go:117] "RemoveContainer" containerID="92ee7defbd0aa16fa9a019c9755fd1f0de62f4484fd5af804c7e31147ac8d53e"
Sep 07 10:30:44.124201 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:44.124016    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8014-140224719722656"
Sep 07 10:30:46.756249 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:46.756210    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:47.071973 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:47.071940    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:47.270735 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:47.270700    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:50.029544 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:50.029508    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:50.126601 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:50.126576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:51.179325 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.179290    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:51.278117 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.278087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:51.658991 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.658939    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:51.758569 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.758537    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:51.861837 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.861814    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"]
Sep 07 10:30:51.862221 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.862207    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95dd921b-958c-4fe1-860f-1bd7dbc33578" containerName="test-curl-8014-140224719722656"
Sep 07 10:30:51.862221 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.862222    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="95dd921b-958c-4fe1-860f-1bd7dbc33578" containerName="test-curl-8014-140224719722656"
Sep 07 10:30:51.862313 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.862291    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="95dd921b-958c-4fe1-860f-1bd7dbc33578" containerName="test-curl-8014-140224719722656"
Sep 07 10:30:51.864786 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.864769    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:51.869120 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.869101    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f957f836-dockercfg-lbbl5\""
Sep 07 10:30:51.869239 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.869151    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f957f836\""
Sep 07 10:30:51.870938 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.870918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecd0447-125f-4341-84fe-c1c820123356-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8\" (UID: \"6ecd0447-125f-4341-84fe-c1c820123356\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:51.871044 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.870958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8sm\" (UniqueName: \"kubernetes.io/projected/6ecd0447-125f-4341-84fe-c1c820123356-kube-api-access-6k8sm\") pod \"payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8\" (UID: \"6ecd0447-125f-4341-84fe-c1c820123356\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:51.885253 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.885229    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"]
Sep 07 10:30:51.917632 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.917560    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:51.972289 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.972262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8sm\" (UniqueName: \"kubernetes.io/projected/6ecd0447-125f-4341-84fe-c1c820123356-kube-api-access-6k8sm\") pod \"payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8\" (UID: \"6ecd0447-125f-4341-84fe-c1c820123356\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:51.972466 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.972446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecd0447-125f-4341-84fe-c1c820123356-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8\" (UID: \"6ecd0447-125f-4341-84fe-c1c820123356\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:51.973114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.973065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecd0447-125f-4341-84fe-c1c820123356-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8\" (UID: \"6ecd0447-125f-4341-84fe-c1c820123356\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:51.982010 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:51.981988    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8sm\" (UniqueName: \"kubernetes.io/projected/6ecd0447-125f-4341-84fe-c1c820123356-kube-api-access-6k8sm\") pod \"payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8\" (UID: \"6ecd0447-125f-4341-84fe-c1c820123356\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:52.085988 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:52.085961    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:52.174535 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:52.174470    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:52.197707 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:52.197676    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:52.444283 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:52.444257    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"]
Sep 07 10:30:52.445990 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:30:52.445959    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecd0447_125f_4341_84fe_c1c820123356.slice/crio-19b6043c3401ad5a453c2768d76593052a719eff1ebe0965befd733b339c570b WatchSource:0}: Error finding container 19b6043c3401ad5a453c2768d76593052a719eff1ebe0965befd733b339c570b: Status 404 returned error can't find the container with id 19b6043c3401ad5a453c2768d76593052a719eff1ebe0965befd733b339c570b
Sep 07 10:30:53.169609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:53.169569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8" event={"ID":"6ecd0447-125f-4341-84fe-c1c820123356","Type":"ContainerStarted","Data":"470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db"}
Sep 07 10:30:53.169780 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:53.169667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8" event={"ID":"6ecd0447-125f-4341-84fe-c1c820123356","Type":"ContainerStarted","Data":"19b6043c3401ad5a453c2768d76593052a719eff1ebe0965befd733b339c570b"}
Sep 07 10:30:53.169780 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:53.169716    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:30:53.188033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:53.187989    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8" podStartSLOduration=2.18797744 podStartE2EDuration="2.18797744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:30:53.185500638 +0000 UTC m=+1159.734015626" watchObservedRunningTime="2026-09-07 10:30:53.18797744 +0000 UTC m=+1159.736492426"
Sep 07 10:30:59.096153 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:59.096051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:30:59.800153 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:59.800122    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:30:59.901026 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:30:59.900994    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:31:03.680821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:03.680782    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:31:03.779967 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:03.779936    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:31:04.176792 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:04.176762    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:31:15.005932 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:15.005893    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:31:28.814204 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:28.814170    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:31:33.103087 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:33.103041    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:31:34.419991 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:34.419950    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:31:39.393338 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.393304    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"]
Sep 07 10:31:39.398079 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.398047    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.401454 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.401432    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f957f836\"/\"kube-root-ca.crt\""
Sep 07 10:31:39.401563 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.401471    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f957f836\"/\"e2e-lifecycle-model-f957f836-kserve-self-signed-certs\""
Sep 07 10:31:39.401563 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.401473    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f957f836\"/\"openshift-service-ca.crt\""
Sep 07 10:31:39.401563 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.401480    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f957f836\"/\"default-dockercfg-mrl45\""
Sep 07 10:31:39.407895 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.407875    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"]
Sep 07 10:31:39.419051 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.419023    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:31:39.484131 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.484103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-home\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.484301 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.484166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tmp-dir\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.484301 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.484203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-dshm\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.484397 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.484296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tls-certs\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.484397 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.484336    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpr49\" (UniqueName: \"kubernetes.io/projected/e0a2c734-79fe-4aab-a3e5-04abfaca8919-kube-api-access-zpr49\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.484397 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.484373    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-model-cache\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585044 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585011    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-home\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585246 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585056    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tmp-dir\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585246 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-dshm\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585378 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585259    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tls-certs\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585378 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpr49\" (UniqueName: \"kubernetes.io/projected/e0a2c734-79fe-4aab-a3e5-04abfaca8919-kube-api-access-zpr49\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585378 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-model-cache\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585539 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tmp-dir\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585539 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-home\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.585686 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.585665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-model-cache\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.587270 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.587253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-dshm\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.587667 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.587652    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tls-certs\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.593665 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.593640    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpr49\" (UniqueName: \"kubernetes.io/projected/e0a2c734-79fe-4aab-a3e5-04abfaca8919-kube-api-access-zpr49\") pod \"e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") " pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.708415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.708313    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:39.977158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:39.977133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"]
Sep 07 10:31:39.979063 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:31:39.979034    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0a2c734_79fe_4aab_a3e5_04abfaca8919.slice/crio-cc8ae357906fe8592b37ca47ef5a93bd37835ed1df8cc0df4e243c108f11c270 WatchSource:0}: Error finding container cc8ae357906fe8592b37ca47ef5a93bd37835ed1df8cc0df4e243c108f11c270: Status 404 returned error can't find the container with id cc8ae357906fe8592b37ca47ef5a93bd37835ed1df8cc0df4e243c108f11c270
Sep 07 10:31:40.389868 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:40.389830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7" event={"ID":"e0a2c734-79fe-4aab-a3e5-04abfaca8919","Type":"ContainerStarted","Data":"a98f741aba0152ecd22ca18a662719133346cc5d21e591688df478fd00c68423"}
Sep 07 10:31:40.389868 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:40.389875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7" event={"ID":"e0a2c734-79fe-4aab-a3e5-04abfaca8919","Type":"ContainerStarted","Data":"cc8ae357906fe8592b37ca47ef5a93bd37835ed1df8cc0df4e243c108f11c270"}
Sep 07 10:31:40.407654 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:40.407608    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7" podStartSLOduration=1.407594349 podStartE2EDuration="1.407594349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:31:40.406793762 +0000 UTC m=+1206.955308749" watchObservedRunningTime="2026-09-07 10:31:40.407594349 +0000 UTC m=+1206.956109380"
Sep 07 10:31:49.708488 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:49.708448    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:49.710965 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:49.708512    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:49.720615 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:49.720595    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:50.450405 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:50.450378    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:31:55.098396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:31:55.098364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:32:14.587965 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:14.587930    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:32:30.923451 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:30.923363    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:32:33.294190 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:33.294148    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:32:46.276217 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.276179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:32:46.305029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.305000    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"]
Sep 07 10:32:46.305335 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.305298    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8" podUID="6ecd0447-125f-4341-84fe-c1c820123356" containerName="payload-pre-processing" containerID="cri-o://470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db" gracePeriod=30
Sep 07 10:32:46.373703 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.373672    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:32:46.474300 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.474265    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:32:46.532472 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.532441    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:32:46.550086 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.550053    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:32:46.566492 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.566455    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecd0447-125f-4341-84fe-c1c820123356-plugins-config-volume\") pod \"6ecd0447-125f-4341-84fe-c1c820123356\" (UID: \"6ecd0447-125f-4341-84fe-c1c820123356\") "
Sep 07 10:32:46.566626 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.566578    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k8sm\" (UniqueName: \"kubernetes.io/projected/6ecd0447-125f-4341-84fe-c1c820123356-kube-api-access-6k8sm\") pod \"6ecd0447-125f-4341-84fe-c1c820123356\" (UID: \"6ecd0447-125f-4341-84fe-c1c820123356\") "
Sep 07 10:32:46.566847 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.566823    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecd0447-125f-4341-84fe-c1c820123356-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ecd0447-125f-4341-84fe-c1c820123356" (UID: "6ecd0447-125f-4341-84fe-c1c820123356"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:32:46.566952 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.566936    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecd0447-125f-4341-84fe-c1c820123356-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:32:46.569121 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.569093    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecd0447-125f-4341-84fe-c1c820123356-kube-api-access-6k8sm" (OuterVolumeSpecName: "kube-api-access-6k8sm") pod "6ecd0447-125f-4341-84fe-c1c820123356" (UID: "6ecd0447-125f-4341-84fe-c1c820123356"). InnerVolumeSpecName "kube-api-access-6k8sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:32:46.667902 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.667868    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8sm\" (UniqueName: \"kubernetes.io/projected/6ecd0447-125f-4341-84fe-c1c820123356-kube-api-access-6k8sm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:32:46.706621 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.706587    2560 generic.go:356] "Generic (PLEG): container finished" podID="6ecd0447-125f-4341-84fe-c1c820123356" containerID="470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db" exitCode=0
Sep 07 10:32:46.706737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.706638    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"
Sep 07 10:32:46.706737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.706675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8" event={"ID":"6ecd0447-125f-4341-84fe-c1c820123356","Type":"ContainerDied","Data":"470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db"}
Sep 07 10:32:46.706737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.706732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8" event={"ID":"6ecd0447-125f-4341-84fe-c1c820123356","Type":"ContainerDied","Data":"19b6043c3401ad5a453c2768d76593052a719eff1ebe0965befd733b339c570b"}
Sep 07 10:32:46.706844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.706760    2560 scope.go:117] "RemoveContainer" containerID="470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db"
Sep 07 10:32:46.715913 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.715896    2560 scope.go:117] "RemoveContainer" containerID="470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db"
Sep 07 10:32:46.716233 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:32:46.716213    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db\": container with ID starting with 470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db not found: ID does not exist" containerID="470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db"
Sep 07 10:32:46.716296 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.716242    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db"} err="failed to get container status \"470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db\": rpc error: code = NotFound desc = could not find container \"470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db\": container with ID starting with 470ed56fab73a918d4c1a614e73c7720f352bcb710bce711211235a47127c3db not found: ID does not exist"
Sep 07 10:32:46.727380 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.727360    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"]
Sep 07 10:32:46.730525 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:46.730502    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f957f836-f59f9cd58-xjfk8"]
Sep 07 10:32:47.895623 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:47.895590    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ecd0447-125f-4341-84fe-c1c820123356" path="/var/lib/kubelet/pods/6ecd0447-125f-4341-84fe-c1c820123356/volumes"
Sep 07 10:32:49.620360 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:49.620324    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:32:52.789836 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:52.789803    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:32:54.597697 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:54.597656    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"]
Sep 07 10:32:54.692831 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:32:54.692793    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:00.890811 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:00.890779    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:09.304499 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.304463    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:33:09.617151 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.617017    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7" podUID="e0a2c734-79fe-4aab-a3e5-04abfaca8919" containerName="main" containerID="cri-o://a98f741aba0152ecd22ca18a662719133346cc5d21e591688df478fd00c68423" gracePeriod=45
Sep 07 10:33:09.818123 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.818087    2560 generic.go:356] "Generic (PLEG): container finished" podID="e0a2c734-79fe-4aab-a3e5-04abfaca8919" containerID="a98f741aba0152ecd22ca18a662719133346cc5d21e591688df478fd00c68423" exitCode=0
Sep 07 10:33:09.818123 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.818107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7" event={"ID":"e0a2c734-79fe-4aab-a3e5-04abfaca8919","Type":"ContainerDied","Data":"a98f741aba0152ecd22ca18a662719133346cc5d21e591688df478fd00c68423"}
Sep 07 10:33:09.868739 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.868686    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:33:09.982572 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.982544    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-home\") pod \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") "
Sep 07 10:33:09.982752 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.982587    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpr49\" (UniqueName: \"kubernetes.io/projected/e0a2c734-79fe-4aab-a3e5-04abfaca8919-kube-api-access-zpr49\") pod \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") "
Sep 07 10:33:09.982752 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.982610    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tls-certs\") pod \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") "
Sep 07 10:33:09.982752 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.982683    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-model-cache\") pod \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") "
Sep 07 10:33:09.982752 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.982710    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-dshm\") pod \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") "
Sep 07 10:33:09.982975 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.982794    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tmp-dir\") pod \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\" (UID: \"e0a2c734-79fe-4aab-a3e5-04abfaca8919\") "
Sep 07 10:33:09.982975 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.982878    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-home" (OuterVolumeSpecName: "home") pod "e0a2c734-79fe-4aab-a3e5-04abfaca8919" (UID: "e0a2c734-79fe-4aab-a3e5-04abfaca8919"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:33:09.982975 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.982929    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0a2c734-79fe-4aab-a3e5-04abfaca8919" (UID: "e0a2c734-79fe-4aab-a3e5-04abfaca8919"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:33:09.983175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.983099    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0a2c734-79fe-4aab-a3e5-04abfaca8919" (UID: "e0a2c734-79fe-4aab-a3e5-04abfaca8919"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:33:09.983224 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.983200    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tmp-dir\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:33:09.983224 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.983220    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-home\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:33:09.983303 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.983234    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-model-cache\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:33:09.984836 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.984814    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-dshm" (OuterVolumeSpecName: "dshm") pod "e0a2c734-79fe-4aab-a3e5-04abfaca8919" (UID: "e0a2c734-79fe-4aab-a3e5-04abfaca8919"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:33:09.984939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.984869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a2c734-79fe-4aab-a3e5-04abfaca8919-kube-api-access-zpr49" (OuterVolumeSpecName: "kube-api-access-zpr49") pod "e0a2c734-79fe-4aab-a3e5-04abfaca8919" (UID: "e0a2c734-79fe-4aab-a3e5-04abfaca8919"). InnerVolumeSpecName "kube-api-access-zpr49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:33:09.984939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:09.984882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0a2c734-79fe-4aab-a3e5-04abfaca8919" (UID: "e0a2c734-79fe-4aab-a3e5-04abfaca8919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:33:10.084179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:10.084051    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpr49\" (UniqueName: \"kubernetes.io/projected/e0a2c734-79fe-4aab-a3e5-04abfaca8919-kube-api-access-zpr49\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:33:10.084179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:10.084161    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0a2c734-79fe-4aab-a3e5-04abfaca8919-tls-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:33:10.084179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:10.084173    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0a2c734-79fe-4aab-a3e5-04abfaca8919-dshm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:33:10.824575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:10.824544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7" event={"ID":"e0a2c734-79fe-4aab-a3e5-04abfaca8919","Type":"ContainerDied","Data":"cc8ae357906fe8592b37ca47ef5a93bd37835ed1df8cc0df4e243c108f11c270"}
Sep 07 10:33:10.824575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:10.824575    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"
Sep 07 10:33:10.825107 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:10.824591    2560 scope.go:117] "RemoveContainer" containerID="a98f741aba0152ecd22ca18a662719133346cc5d21e591688df478fd00c68423"
Sep 07 10:33:10.845998 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:10.845970    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"]
Sep 07 10:33:10.854341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:10.854318    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f957f836/e2e-lifecycle-model-f957f836-kserve-59cf66b67f-l2dn7"]
Sep 07 10:33:11.895920 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:11.895887    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a2c734-79fe-4aab-a3e5-04abfaca8919" path="/var/lib/kubelet/pods/e0a2c734-79fe-4aab-a3e5-04abfaca8919/volumes"
Sep 07 10:33:25.390563 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:25.390530    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:33:31.209582 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:31.209543    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:31.646923 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:31.646887    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:37.489691 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:37.489652    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:37.556056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:37.556018    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:37.731741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:37.731702    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:38.957647 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:38.957618    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:39.057802 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:39.057766    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:39.159270 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:39.159233    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:39.735825 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:39.735795    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:39.827879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:39.827841    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:39.926516 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:39.926482    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:41.113352 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:41.113315    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:44.191011 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:44.190977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:33:46.906439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:46.906402    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:46.978321 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:46.978287    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:47.565666 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:47.565635    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:51.572833 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:51.572792    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"]
Sep 07 10:33:51.593829 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:51.593802    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:52.622840 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:52.622806    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:53.251647 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:53.251613    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:53.351341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:53.351298    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:53.448761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:53.448730    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:33:56.422754 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:33:56.422664    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:02.094083 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:02.094045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:34:03.898972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:03.898926    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:04.128526 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:04.128474    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:06.592264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.592226    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8" podUID="5fe04cb0-dc93-472d-9565-63ed960abffb" containerName="main" containerID="cri-o://94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5" gracePeriod=45
Sep 07 10:34:06.833593 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.833565    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:34:06.969645 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.969567    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl2r2\" (UniqueName: \"kubernetes.io/projected/5fe04cb0-dc93-472d-9565-63ed960abffb-kube-api-access-jl2r2\") pod \"5fe04cb0-dc93-472d-9565-63ed960abffb\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") "
Sep 07 10:34:06.969645 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.969601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-dshm\") pod \"5fe04cb0-dc93-472d-9565-63ed960abffb\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") "
Sep 07 10:34:06.969645 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.969640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe04cb0-dc93-472d-9565-63ed960abffb-tls-certs\") pod \"5fe04cb0-dc93-472d-9565-63ed960abffb\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") "
Sep 07 10:34:06.969922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.969663    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-home\") pod \"5fe04cb0-dc93-472d-9565-63ed960abffb\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") "
Sep 07 10:34:06.969922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.969743    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-model-cache\") pod \"5fe04cb0-dc93-472d-9565-63ed960abffb\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") "
Sep 07 10:34:06.969922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.969810    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-tmp-dir\") pod \"5fe04cb0-dc93-472d-9565-63ed960abffb\" (UID: \"5fe04cb0-dc93-472d-9565-63ed960abffb\") "
Sep 07 10:34:06.969922 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.969901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-home" (OuterVolumeSpecName: "home") pod "5fe04cb0-dc93-472d-9565-63ed960abffb" (UID: "5fe04cb0-dc93-472d-9565-63ed960abffb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:34:06.970176 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.969966    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fe04cb0-dc93-472d-9565-63ed960abffb" (UID: "5fe04cb0-dc93-472d-9565-63ed960abffb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:34:06.970176 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.970087    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fe04cb0-dc93-472d-9565-63ed960abffb" (UID: "5fe04cb0-dc93-472d-9565-63ed960abffb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:34:06.970176 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.970166    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-home\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:34:06.970345 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.970184    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-model-cache\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:34:06.970345 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.970193    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-tmp-dir\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:34:06.971856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.971830    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-dshm" (OuterVolumeSpecName: "dshm") pod "5fe04cb0-dc93-472d-9565-63ed960abffb" (UID: "5fe04cb0-dc93-472d-9565-63ed960abffb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:34:06.971962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.971878    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe04cb0-dc93-472d-9565-63ed960abffb-kube-api-access-jl2r2" (OuterVolumeSpecName: "kube-api-access-jl2r2") pod "5fe04cb0-dc93-472d-9565-63ed960abffb" (UID: "5fe04cb0-dc93-472d-9565-63ed960abffb"). InnerVolumeSpecName "kube-api-access-jl2r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:34:06.971962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:06.971896    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe04cb0-dc93-472d-9565-63ed960abffb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fe04cb0-dc93-472d-9565-63ed960abffb" (UID: "5fe04cb0-dc93-472d-9565-63ed960abffb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:34:07.071100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.071060    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2r2\" (UniqueName: \"kubernetes.io/projected/5fe04cb0-dc93-472d-9565-63ed960abffb-kube-api-access-jl2r2\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:34:07.071100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.071098    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe04cb0-dc93-472d-9565-63ed960abffb-dshm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:34:07.071264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.071116    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe04cb0-dc93-472d-9565-63ed960abffb-tls-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:34:07.099492 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.099468    2560 generic.go:356] "Generic (PLEG): container finished" podID="5fe04cb0-dc93-472d-9565-63ed960abffb" containerID="94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5" exitCode=0
Sep 07 10:34:07.099638 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.099520    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"
Sep 07 10:34:07.099638 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.099561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8" event={"ID":"5fe04cb0-dc93-472d-9565-63ed960abffb","Type":"ContainerDied","Data":"94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5"}
Sep 07 10:34:07.099638 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.099604    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8" event={"ID":"5fe04cb0-dc93-472d-9565-63ed960abffb","Type":"ContainerDied","Data":"dd1906ca49d529628eb896260db2eab4a525784fc17b4fe927728b13d642d43b"}
Sep 07 10:34:07.099638 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.099621    2560 scope.go:117] "RemoveContainer" containerID="94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5"
Sep 07 10:34:07.108609 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.108590    2560 scope.go:117] "RemoveContainer" containerID="94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5"
Sep 07 10:34:07.108894 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:34:07.108873    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5\": container with ID starting with 94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5 not found: ID does not exist" containerID="94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5"
Sep 07 10:34:07.108955 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.108908    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5"} err="failed to get container status \"94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5\": rpc error: code = NotFound desc = could not find container \"94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5\": container with ID starting with 94d4d74a1cdb915bf04018b532bb548bd4a67bf923d8c43fe001c42952abd6c5 not found: ID does not exist"
Sep 07 10:34:07.126421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.126393    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"]
Sep 07 10:34:07.130194 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.130171    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/auth-test-model-0a577c-kserve-6b945dcfcf-p8vh8"]
Sep 07 10:34:07.895252 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:07.895220    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe04cb0-dc93-472d-9565-63ed960abffb" path="/var/lib/kubelet/pods/5fe04cb0-dc93-472d-9565-63ed960abffb/volumes"
Sep 07 10:34:13.284269 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:13.284236    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:18.977929 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:18.977886    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:25.067667 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:25.067629    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:31.033523 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:31.033489    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:31.135039 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:31.135010    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:31.367586 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:31.367513    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:31.669163 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:31.669082    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:31.723122 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:31.723091    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:34.382561 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:34.382494    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:34.481492 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:34.481466    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:35.609259 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.609228    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"]
Sep 07 10:34:35.609745 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.609723    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecd0447-125f-4341-84fe-c1c820123356" containerName="payload-pre-processing"
Sep 07 10:34:35.609904 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.609822    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecd0447-125f-4341-84fe-c1c820123356" containerName="payload-pre-processing"
Sep 07 10:34:35.609904 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.609850    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe04cb0-dc93-472d-9565-63ed960abffb" containerName="main"
Sep 07 10:34:35.609904 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.609859    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe04cb0-dc93-472d-9565-63ed960abffb" containerName="main"
Sep 07 10:34:35.609904 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.609886    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a2c734-79fe-4aab-a3e5-04abfaca8919" containerName="main"
Sep 07 10:34:35.609904 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.609893    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a2c734-79fe-4aab-a3e5-04abfaca8919" containerName="main"
Sep 07 10:34:35.610183 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.610013    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a2c734-79fe-4aab-a3e5-04abfaca8919" containerName="main"
Sep 07 10:34:35.610183 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.610025    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe04cb0-dc93-472d-9565-63ed960abffb" containerName="main"
Sep 07 10:34:35.610183 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.610036    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ecd0447-125f-4341-84fe-c1c820123356" containerName="payload-pre-processing"
Sep 07 10:34:35.614415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.614395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.616621 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.616602    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fbea94-serving-cert\""
Sep 07 10:34:35.622506 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.622482    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"]
Sep 07 10:34:35.640277 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.640251    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:35.703616 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.703587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-maas-api-tls\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.703731 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.703638    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzjd\" (UniqueName: \"kubernetes.io/projected/66e4a62c-b615-405e-9743-9632463c7302-kube-api-access-8vzjd\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.703731 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.703661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e4a62c-b615-405e-9743-9632463c7302-openshift-service-ca\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.703809 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.703771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-metrics-certs\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.749573 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.749545    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:35.804870 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.804846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-metrics-certs\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.804962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.804874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-maas-api-tls\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.804962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.804915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzjd\" (UniqueName: \"kubernetes.io/projected/66e4a62c-b615-405e-9743-9632463c7302-kube-api-access-8vzjd\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.804962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.804948    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e4a62c-b615-405e-9743-9632463c7302-openshift-service-ca\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.805633 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.805611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e4a62c-b615-405e-9743-9632463c7302-openshift-service-ca\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.807380 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.807362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-metrics-certs\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.807463 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.807383    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-maas-api-tls\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.820050 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.820028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzjd\" (UniqueName: \"kubernetes.io/projected/66e4a62c-b615-405e-9743-9632463c7302-kube-api-access-8vzjd\") pod \"maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:35.890753 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.890690    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:35.928385 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:35.928363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:36.123724 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.123694    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:36.198603 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.198580    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"]
Sep 07 10:34:36.200731 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:34:36.200700    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e4a62c_b615_405e_9743_9632463c7302.slice/crio-e616871cc6625f72d94174ea2d753c4029d9c76d123d8752a8245a602bc61479 WatchSource:0}: Error finding container e616871cc6625f72d94174ea2d753c4029d9c76d123d8752a8245a602bc61479: Status 404 returned error can't find the container with id e616871cc6625f72d94174ea2d753c4029d9c76d123d8752a8245a602bc61479
Sep 07 10:34:36.201096 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.201061    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:36.202105 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.202064    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:34:36.230040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.230012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn" event={"ID":"66e4a62c-b615-405e-9743-9632463c7302","Type":"ContainerStarted","Data":"e616871cc6625f72d94174ea2d753c4029d9c76d123d8752a8245a602bc61479"}
Sep 07 10:34:36.272728 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.272703    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"]
Sep 07 10:34:36.277163 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.277147    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:36.279708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.279689    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fbea94-dockercfg-ld5zd\""
Sep 07 10:34:36.279875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.279860    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fbea94\""
Sep 07 10:34:36.285465 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.285436    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"]
Sep 07 10:34:36.308048 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.308013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r58fw\" (UniqueName: \"kubernetes.io/projected/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-kube-api-access-r58fw\") pod \"payload-processing-e2e-api-a-fbea94-656976667c-dwhnt\" (UID: \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:36.308187 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.308082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fbea94-656976667c-dwhnt\" (UID: \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:36.409329 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.409296    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r58fw\" (UniqueName: \"kubernetes.io/projected/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-kube-api-access-r58fw\") pod \"payload-processing-e2e-api-a-fbea94-656976667c-dwhnt\" (UID: \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:36.409479 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.409397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fbea94-656976667c-dwhnt\" (UID: \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:36.410039 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.410016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fbea94-656976667c-dwhnt\" (UID: \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:36.417023 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.417005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r58fw\" (UniqueName: \"kubernetes.io/projected/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-kube-api-access-r58fw\") pod \"payload-processing-e2e-api-a-fbea94-656976667c-dwhnt\" (UID: \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\") " pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:36.588799 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.588769    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:36.862009 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:36.861981    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"]
Sep 07 10:34:36.863868 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:34:36.863838    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0882d6c3_4f2a_4c0e_b456_0c6e777a2417.slice/crio-41dd15c77ea6445cf4e26352172ba4a4fbe3a04776825814d598b7fb1f1bdbe3 WatchSource:0}: Error finding container 41dd15c77ea6445cf4e26352172ba4a4fbe3a04776825814d598b7fb1f1bdbe3: Status 404 returned error can't find the container with id 41dd15c77ea6445cf4e26352172ba4a4fbe3a04776825814d598b7fb1f1bdbe3
Sep 07 10:34:37.237225 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:37.237138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt" event={"ID":"0882d6c3-4f2a-4c0e-b456-0c6e777a2417","Type":"ContainerStarted","Data":"b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca"}
Sep 07 10:34:37.237225 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:37.237192    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt" event={"ID":"0882d6c3-4f2a-4c0e-b456-0c6e777a2417","Type":"ContainerStarted","Data":"41dd15c77ea6445cf4e26352172ba4a4fbe3a04776825814d598b7fb1f1bdbe3"}
Sep 07 10:34:37.237444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:37.237334    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:37.238711 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:37.238679    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn" event={"ID":"66e4a62c-b615-405e-9743-9632463c7302","Type":"ContainerStarted","Data":"d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3"}
Sep 07 10:34:37.238835 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:37.238797    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:37.258148 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:37.258102    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt" podStartSLOduration=1.2580884829999999 podStartE2EDuration="1.258088483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:34:37.256731402 +0000 UTC m=+1383.805246393" watchObservedRunningTime="2026-09-07 10:34:37.258088483 +0000 UTC m=+1383.806603461"
Sep 07 10:34:37.280184 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:37.280140    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn" podStartSLOduration=1.705737268 podStartE2EDuration="2.280126406s" totalImagesPullingTime="574.389138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:34:37.27829044 +0000 UTC m=+1383.826805428" watchObservedRunningTime="2026-09-07 10:34:37.280126406 +0000 UTC m=+1383.828641393"
Sep 07 10:34:38.485546 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:38.485512    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:43.248887 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:43.248856    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:34:43.304134 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:43.304100    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:34:47.246554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:47.246522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:47.523349 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:47.523271    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:48.245219 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:48.245192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:34:56.509122 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:56.509084    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:59.326244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:59.326201    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:34:59.769814 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:34:59.769733    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:00.088593 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:00.088557    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:00.503616 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:00.503531    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:03.587410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:03.587372    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:03.684368 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:03.684338    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:03.785166 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:03.785140    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:03.916040 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:03.915965    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:35:04.775335 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:04.775298    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:04.953525 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:04.953486    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:05.054271 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:05.054240    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:05.393279 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:05.393195    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:05.503670 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:05.503638    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:19.899400 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:19.899366    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:35:28.509249 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:28.509166    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:30.328042 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:30.328014    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:31.356420 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:31.356391    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:39.207997 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:39.207962    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:35:42.589923 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:42.589889    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-8017-d12aca"]
Sep 07 10:35:42.593575 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:42.593552    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-d12aca"
Sep 07 10:35:42.596177 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:42.596154    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qpjjb\""
Sep 07 10:35:42.596974 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:42.596950    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8017-d12aca"]
Sep 07 10:35:42.659541 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:42.659522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgzq\" (UniqueName: \"kubernetes.io/projected/4d5e22c8-eb39-4034-85cf-19a1c10e0e03-kube-api-access-fqgzq\") pod \"mt-curl-8017-d12aca\" (UID: \"4d5e22c8-eb39-4034-85cf-19a1c10e0e03\") " pod="opendatahub/mt-curl-8017-d12aca"
Sep 07 10:35:42.760100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:42.760058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgzq\" (UniqueName: \"kubernetes.io/projected/4d5e22c8-eb39-4034-85cf-19a1c10e0e03-kube-api-access-fqgzq\") pod \"mt-curl-8017-d12aca\" (UID: \"4d5e22c8-eb39-4034-85cf-19a1c10e0e03\") " pod="opendatahub/mt-curl-8017-d12aca"
Sep 07 10:35:42.768722 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:42.768694    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgzq\" (UniqueName: \"kubernetes.io/projected/4d5e22c8-eb39-4034-85cf-19a1c10e0e03-kube-api-access-fqgzq\") pod \"mt-curl-8017-d12aca\" (UID: \"4d5e22c8-eb39-4034-85cf-19a1c10e0e03\") " pod="opendatahub/mt-curl-8017-d12aca"
Sep 07 10:35:42.904637 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:42.904553    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-d12aca"
Sep 07 10:35:43.174031 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:43.173956    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8017-d12aca"]
Sep 07 10:35:43.177002 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:35:43.176972    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5e22c8_eb39_4034_85cf_19a1c10e0e03.slice/crio-6742f4b04ed9ffb0c868f91117490eba3bf5731aa887bb64d836abb9f0983e29 WatchSource:0}: Error finding container 6742f4b04ed9ffb0c868f91117490eba3bf5731aa887bb64d836abb9f0983e29: Status 404 returned error can't find the container with id 6742f4b04ed9ffb0c868f91117490eba3bf5731aa887bb64d836abb9f0983e29
Sep 07 10:35:43.548286 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:43.548250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8017-d12aca" event={"ID":"4d5e22c8-eb39-4034-85cf-19a1c10e0e03","Type":"ContainerStarted","Data":"6742f4b04ed9ffb0c868f91117490eba3bf5731aa887bb64d836abb9f0983e29"}
Sep 07 10:35:44.554146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:44.554113    2560 generic.go:356] "Generic (PLEG): container finished" podID="4d5e22c8-eb39-4034-85cf-19a1c10e0e03" containerID="25fbb7cdf3b46343b52d4fc1bea7c4474d8e01b9671c333656b3ce0cda348d34" exitCode=0
Sep 07 10:35:44.554574 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:44.554193    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8017-d12aca" event={"ID":"4d5e22c8-eb39-4034-85cf-19a1c10e0e03","Type":"ContainerDied","Data":"25fbb7cdf3b46343b52d4fc1bea7c4474d8e01b9671c333656b3ce0cda348d34"}
Sep 07 10:35:45.684725 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.684701    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-d12aca"
Sep 07 10:35:45.727979 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.727954    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-8017-d12aca_4d5e22c8-eb39-4034-85cf-19a1c10e0e03/mt-curl-8017-d12aca/0.log"
Sep 07 10:35:45.766246 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.766217    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-8017-d12aca"]
Sep 07 10:35:45.772022 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.772002    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-8017-d12aca"]
Sep 07 10:35:45.782619 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.782598    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqgzq\" (UniqueName: \"kubernetes.io/projected/4d5e22c8-eb39-4034-85cf-19a1c10e0e03-kube-api-access-fqgzq\") pod \"4d5e22c8-eb39-4034-85cf-19a1c10e0e03\" (UID: \"4d5e22c8-eb39-4034-85cf-19a1c10e0e03\") "
Sep 07 10:35:45.784599 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.784573    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5e22c8-eb39-4034-85cf-19a1c10e0e03-kube-api-access-fqgzq" (OuterVolumeSpecName: "kube-api-access-fqgzq") pod "4d5e22c8-eb39-4034-85cf-19a1c10e0e03" (UID: "4d5e22c8-eb39-4034-85cf-19a1c10e0e03"). InnerVolumeSpecName "kube-api-access-fqgzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:35:45.883889 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.883830    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqgzq\" (UniqueName: \"kubernetes.io/projected/4d5e22c8-eb39-4034-85cf-19a1c10e0e03-kube-api-access-fqgzq\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:35:45.896081 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.896039    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5e22c8-eb39-4034-85cf-19a1c10e0e03" path="/var/lib/kubelet/pods/4d5e22c8-eb39-4034-85cf-19a1c10e0e03/volumes"
Sep 07 10:35:45.898340 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.898322    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-8017-0fe254"]
Sep 07 10:35:45.898696 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.898684    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5e22c8-eb39-4034-85cf-19a1c10e0e03" containerName="mt-curl-8017-d12aca"
Sep 07 10:35:45.898735 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.898697    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5e22c8-eb39-4034-85cf-19a1c10e0e03" containerName="mt-curl-8017-d12aca"
Sep 07 10:35:45.898779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.898764    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5e22c8-eb39-4034-85cf-19a1c10e0e03" containerName="mt-curl-8017-d12aca"
Sep 07 10:35:45.901875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.901862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-0fe254"
Sep 07 10:35:45.912982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.912960    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8017-0fe254"]
Sep 07 10:35:45.984444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:45.984418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sl8l\" (UniqueName: \"kubernetes.io/projected/217066d1-b222-47d6-96e0-6f78dc0c9e8f-kube-api-access-7sl8l\") pod \"mt-curl-8017-0fe254\" (UID: \"217066d1-b222-47d6-96e0-6f78dc0c9e8f\") " pod="opendatahub/mt-curl-8017-0fe254"
Sep 07 10:35:46.085708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.085686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sl8l\" (UniqueName: \"kubernetes.io/projected/217066d1-b222-47d6-96e0-6f78dc0c9e8f-kube-api-access-7sl8l\") pod \"mt-curl-8017-0fe254\" (UID: \"217066d1-b222-47d6-96e0-6f78dc0c9e8f\") " pod="opendatahub/mt-curl-8017-0fe254"
Sep 07 10:35:46.095968 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.095949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sl8l\" (UniqueName: \"kubernetes.io/projected/217066d1-b222-47d6-96e0-6f78dc0c9e8f-kube-api-access-7sl8l\") pod \"mt-curl-8017-0fe254\" (UID: \"217066d1-b222-47d6-96e0-6f78dc0c9e8f\") " pod="opendatahub/mt-curl-8017-0fe254"
Sep 07 10:35:46.212110 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.212021    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-0fe254"
Sep 07 10:35:46.479461 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.479439    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8017-0fe254"]
Sep 07 10:35:46.481829 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:35:46.481803    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod217066d1_b222_47d6_96e0_6f78dc0c9e8f.slice/crio-d536ae7fd9a6093c0d2df3ca260c6431e6b47d7a404d1abaf9a10c1e8f720455 WatchSource:0}: Error finding container d536ae7fd9a6093c0d2df3ca260c6431e6b47d7a404d1abaf9a10c1e8f720455: Status 404 returned error can't find the container with id d536ae7fd9a6093c0d2df3ca260c6431e6b47d7a404d1abaf9a10c1e8f720455
Sep 07 10:35:46.564443 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.564413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8017-0fe254" event={"ID":"217066d1-b222-47d6-96e0-6f78dc0c9e8f","Type":"ContainerStarted","Data":"d536ae7fd9a6093c0d2df3ca260c6431e6b47d7a404d1abaf9a10c1e8f720455"}
Sep 07 10:35:46.565673 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.565656    2560 scope.go:117] "RemoveContainer" containerID="25fbb7cdf3b46343b52d4fc1bea7c4474d8e01b9671c333656b3ce0cda348d34"
Sep 07 10:35:46.565778 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.565679    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-d12aca"
Sep 07 10:35:46.867314 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.867288    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"]
Sep 07 10:35:46.867680 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.867518    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn" podUID="66e4a62c-b615-405e-9743-9632463c7302" containerName="maas-api" containerID="cri-o://d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3" gracePeriod=30
Sep 07 10:35:46.893821 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.893793    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:46.911830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.911805    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"]
Sep 07 10:35:46.912056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.912028    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt" podUID="0882d6c3-4f2a-4c0e-b456-0c6e777a2417" containerName="payload-processing" containerID="cri-o://b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca" gracePeriod=30
Sep 07 10:35:46.995296 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:46.995226    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:47.095562 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.095529    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:47.158043 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.158008    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:47.158290 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.158269    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:35:47.168971 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.168953    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:35:47.194687 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.194633    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e4a62c-b615-405e-9743-9632463c7302-openshift-service-ca\") pod \"66e4a62c-b615-405e-9743-9632463c7302\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") "
Sep 07 10:35:47.194993 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.194719    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-maas-api-tls\") pod \"66e4a62c-b615-405e-9743-9632463c7302\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") "
Sep 07 10:35:47.195179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.195156    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e4a62c-b615-405e-9743-9632463c7302-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66e4a62c-b615-405e-9743-9632463c7302" (UID: "66e4a62c-b615-405e-9743-9632463c7302"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:35:47.195298 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.195282    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-plugins-config-volume\") pod \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\" (UID: \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\") "
Sep 07 10:35:47.195349 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.195326    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vzjd\" (UniqueName: \"kubernetes.io/projected/66e4a62c-b615-405e-9743-9632463c7302-kube-api-access-8vzjd\") pod \"66e4a62c-b615-405e-9743-9632463c7302\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") "
Sep 07 10:35:47.195399 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.195353    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-metrics-certs\") pod \"66e4a62c-b615-405e-9743-9632463c7302\" (UID: \"66e4a62c-b615-405e-9743-9632463c7302\") "
Sep 07 10:35:47.195399 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.195379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r58fw\" (UniqueName: \"kubernetes.io/projected/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-kube-api-access-r58fw\") pod \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\" (UID: \"0882d6c3-4f2a-4c0e-b456-0c6e777a2417\") "
Sep 07 10:35:47.195693 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.195677    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66e4a62c-b615-405e-9743-9632463c7302-openshift-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:35:47.196461 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.196418    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0882d6c3-4f2a-4c0e-b456-0c6e777a2417" (UID: "0882d6c3-4f2a-4c0e-b456-0c6e777a2417"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:35:47.197958 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.197937    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66e4a62c-b615-405e-9743-9632463c7302" (UID: "66e4a62c-b615-405e-9743-9632463c7302"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:35:47.198043 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.197951    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e4a62c-b615-405e-9743-9632463c7302-kube-api-access-8vzjd" (OuterVolumeSpecName: "kube-api-access-8vzjd") pod "66e4a62c-b615-405e-9743-9632463c7302" (UID: "66e4a62c-b615-405e-9743-9632463c7302"). InnerVolumeSpecName "kube-api-access-8vzjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:35:47.198043 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.197973    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-kube-api-access-r58fw" (OuterVolumeSpecName: "kube-api-access-r58fw") pod "0882d6c3-4f2a-4c0e-b456-0c6e777a2417" (UID: "0882d6c3-4f2a-4c0e-b456-0c6e777a2417"). InnerVolumeSpecName "kube-api-access-r58fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:35:47.198708 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.198691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66e4a62c-b615-405e-9743-9632463c7302" (UID: "66e4a62c-b615-405e-9743-9632463c7302"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:35:47.296681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.296657    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-maas-api-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:35:47.296681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.296678    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:35:47.296818 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.296688    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vzjd\" (UniqueName: \"kubernetes.io/projected/66e4a62c-b615-405e-9743-9632463c7302-kube-api-access-8vzjd\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:35:47.296818 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.296701    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66e4a62c-b615-405e-9743-9632463c7302-metrics-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:35:47.296818 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.296711    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r58fw\" (UniqueName: \"kubernetes.io/projected/0882d6c3-4f2a-4c0e-b456-0c6e777a2417-kube-api-access-r58fw\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:35:47.425742 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.425717    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:47.572641 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.572559    2560 generic.go:356] "Generic (PLEG): container finished" podID="0882d6c3-4f2a-4c0e-b456-0c6e777a2417" containerID="b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca" exitCode=0
Sep 07 10:35:47.572641 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.572598    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"
Sep 07 10:35:47.572848 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.572643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt" event={"ID":"0882d6c3-4f2a-4c0e-b456-0c6e777a2417","Type":"ContainerDied","Data":"b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca"}
Sep 07 10:35:47.572848 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.572700    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt" event={"ID":"0882d6c3-4f2a-4c0e-b456-0c6e777a2417","Type":"ContainerDied","Data":"41dd15c77ea6445cf4e26352172ba4a4fbe3a04776825814d598b7fb1f1bdbe3"}
Sep 07 10:35:47.572848 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.572730    2560 scope.go:117] "RemoveContainer" containerID="b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca"
Sep 07 10:35:47.574055 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.574033    2560 generic.go:356] "Generic (PLEG): container finished" podID="66e4a62c-b615-405e-9743-9632463c7302" containerID="d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3" exitCode=0
Sep 07 10:35:47.574168 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.574138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn" event={"ID":"66e4a62c-b615-405e-9743-9632463c7302","Type":"ContainerDied","Data":"d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3"}
Sep 07 10:35:47.574168 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.574148    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"
Sep 07 10:35:47.574285 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.574179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn" event={"ID":"66e4a62c-b615-405e-9743-9632463c7302","Type":"ContainerDied","Data":"e616871cc6625f72d94174ea2d753c4029d9c76d123d8752a8245a602bc61479"}
Sep 07 10:35:47.575804 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.575781    2560 generic.go:356] "Generic (PLEG): container finished" podID="217066d1-b222-47d6-96e0-6f78dc0c9e8f" containerID="afb26f0e090bcff6dca972dd8def585fac702484ea8583754c4ef2b796ecd76f" exitCode=0
Sep 07 10:35:47.575914 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.575803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8017-0fe254" event={"ID":"217066d1-b222-47d6-96e0-6f78dc0c9e8f","Type":"ContainerDied","Data":"afb26f0e090bcff6dca972dd8def585fac702484ea8583754c4ef2b796ecd76f"}
Sep 07 10:35:47.581795 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.581775    2560 scope.go:117] "RemoveContainer" containerID="b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca"
Sep 07 10:35:47.582250 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:35:47.582093    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca\": container with ID starting with b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca not found: ID does not exist" containerID="b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca"
Sep 07 10:35:47.582250 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.582122    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca"} err="failed to get container status \"b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca\": rpc error: code = NotFound desc = could not find container \"b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca\": container with ID starting with b9095b797a3b13ea3c3ffe85ca0efcc17ad043b649ae23a5db45a694db87beca not found: ID does not exist"
Sep 07 10:35:47.582250 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.582139    2560 scope.go:117] "RemoveContainer" containerID="d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3"
Sep 07 10:35:47.590999 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.590984    2560 scope.go:117] "RemoveContainer" containerID="d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3"
Sep 07 10:35:47.591275 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:35:47.591258    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3\": container with ID starting with d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3 not found: ID does not exist" containerID="d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3"
Sep 07 10:35:47.591325 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.591283    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3"} err="failed to get container status \"d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3\": rpc error: code = NotFound desc = could not find container \"d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3\": container with ID starting with d1883f1ffe6f66210d8d39013a0dab5582f19ae31424d73644e32576ba1e3be3 not found: ID does not exist"
Sep 07 10:35:47.607797 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.607775    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"]
Sep 07 10:35:47.612855 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.612833    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fbea94-6b78c98c44-6f6jn"]
Sep 07 10:35:47.625311 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.625286    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"]
Sep 07 10:35:47.630796 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.630775    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fbea94-656976667c-dwhnt"]
Sep 07 10:35:47.895819 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.895742    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0882d6c3-4f2a-4c0e-b456-0c6e777a2417" path="/var/lib/kubelet/pods/0882d6c3-4f2a-4c0e-b456-0c6e777a2417/volumes"
Sep 07 10:35:47.896186 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:47.896132    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66e4a62c-b615-405e-9743-9632463c7302" path="/var/lib/kubelet/pods/66e4a62c-b615-405e-9743-9632463c7302/volumes"
Sep 07 10:35:48.726183 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:48.726161    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-0fe254"
Sep 07 10:35:48.750049 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:48.750013    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-8017-0fe254_217066d1-b222-47d6-96e0-6f78dc0c9e8f/mt-curl-8017-0fe254/0.log"
Sep 07 10:35:48.780348 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:48.780318    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-8017-0fe254"]
Sep 07 10:35:48.783162 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:48.783142    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-8017-0fe254"]
Sep 07 10:35:48.809843 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:48.809819    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sl8l\" (UniqueName: \"kubernetes.io/projected/217066d1-b222-47d6-96e0-6f78dc0c9e8f-kube-api-access-7sl8l\") pod \"217066d1-b222-47d6-96e0-6f78dc0c9e8f\" (UID: \"217066d1-b222-47d6-96e0-6f78dc0c9e8f\") "
Sep 07 10:35:48.811786 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:48.811754    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217066d1-b222-47d6-96e0-6f78dc0c9e8f-kube-api-access-7sl8l" (OuterVolumeSpecName: "kube-api-access-7sl8l") pod "217066d1-b222-47d6-96e0-6f78dc0c9e8f" (UID: "217066d1-b222-47d6-96e0-6f78dc0c9e8f"). InnerVolumeSpecName "kube-api-access-7sl8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:35:48.910770 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:48.910717    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sl8l\" (UniqueName: \"kubernetes.io/projected/217066d1-b222-47d6-96e0-6f78dc0c9e8f-kube-api-access-7sl8l\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:35:49.588935 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:49.588906    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d536ae7fd9a6093c0d2df3ca260c6431e6b47d7a404d1abaf9a10c1e8f720455"
Sep 07 10:35:49.588935 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:49.588914    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8017-0fe254"
Sep 07 10:35:49.598692 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:35:49.598666    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-8017-0fe254\" is forbidden: User \"system:node:ip-10-0-128-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-191.ec2.internal' and this object" podUID="217066d1-b222-47d6-96e0-6f78dc0c9e8f" pod="opendatahub/mt-curl-8017-0fe254"
Sep 07 10:35:49.896622 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:49.896528    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217066d1-b222-47d6-96e0-6f78dc0c9e8f" path="/var/lib/kubelet/pods/217066d1-b222-47d6-96e0-6f78dc0c9e8f/volumes"
Sep 07 10:35:50.807663 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:50.807624    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:51.936851 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:51.936806    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:53.563899 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:53.563861    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:53.661395 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:53.661363    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:53.846307 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:53.846199    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:54.608147 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:54.608110    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:35:54.944546 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:54.944463    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:35:59.560485 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:35:59.560446    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:02.507352 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.507311    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"]
Sep 07 10:36:02.507911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.507864    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217066d1-b222-47d6-96e0-6f78dc0c9e8f" containerName="mt-curl-8017-0fe254"
Sep 07 10:36:02.507911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.507883    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="217066d1-b222-47d6-96e0-6f78dc0c9e8f" containerName="mt-curl-8017-0fe254"
Sep 07 10:36:02.507911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.507896    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66e4a62c-b615-405e-9743-9632463c7302" containerName="maas-api"
Sep 07 10:36:02.507911 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.507905    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="66e4a62c-b615-405e-9743-9632463c7302" containerName="maas-api"
Sep 07 10:36:02.508162 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.507924    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0882d6c3-4f2a-4c0e-b456-0c6e777a2417" containerName="payload-processing"
Sep 07 10:36:02.508162 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.507933    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0882d6c3-4f2a-4c0e-b456-0c6e777a2417" containerName="payload-processing"
Sep 07 10:36:02.508162 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.508024    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="217066d1-b222-47d6-96e0-6f78dc0c9e8f" containerName="mt-curl-8017-0fe254"
Sep 07 10:36:02.508162 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.508038    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0882d6c3-4f2a-4c0e-b456-0c6e777a2417" containerName="payload-processing"
Sep 07 10:36:02.508162 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.508049    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="66e4a62c-b615-405e-9743-9632463c7302" containerName="maas-api"
Sep 07 10:36:02.510883 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.510862    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.513727 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.513707    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"openshift-service-ca.crt\""
Sep 07 10:36:02.513832 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.513736    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"default-dockercfg-c5g9z\""
Sep 07 10:36:02.514973 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.514955    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"rate-test-model-0a577c-kserve-self-signed-certs\""
Sep 07 10:36:02.515092 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.515049    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-0a577c\"/\"kube-root-ca.crt\""
Sep 07 10:36:02.522023 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.522004    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"]
Sep 07 10:36:02.603432 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.603403    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:02.640260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.640230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tls-certs\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.640421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.640264    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-home\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.640421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.640298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-dshm\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.640421 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.640397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tmp-dir\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.640570 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.640431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2t27\" (UniqueName: \"kubernetes.io/projected/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-kube-api-access-x2t27\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.640570 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.640461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-model-cache\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.741868 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.741826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tls-certs\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.741868 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.741868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-home\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.742145 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.741896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-dshm\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.742145 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.741946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tmp-dir\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.742145 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.741975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2t27\" (UniqueName: \"kubernetes.io/projected/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-kube-api-access-x2t27\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.742145 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.742113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-model-cache\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.742361 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.742309    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-home\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.742417 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.742368    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tmp-dir\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.742482 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.742460    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-model-cache\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.744257 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.744234    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-dshm\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.744459 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.744443    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tls-certs\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.752318 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.752299    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2t27\" (UniqueName: \"kubernetes.io/projected/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-kube-api-access-x2t27\") pod \"rate-test-model-0a577c-kserve-db7785679-zl5mx\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") " pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:02.819634 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:02.819608    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:03.091045 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:03.091013    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"]
Sep 07 10:36:03.093374 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:36:03.093345    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c5d87ec_03d1_4f85_98c1_9389de56dc9d.slice/crio-cb94ddd66816a200f6e13ac6ddfdc2d32bd89a66c2e7100e5b9354ac5a6e454e WatchSource:0}: Error finding container cb94ddd66816a200f6e13ac6ddfdc2d32bd89a66c2e7100e5b9354ac5a6e454e: Status 404 returned error can't find the container with id cb94ddd66816a200f6e13ac6ddfdc2d32bd89a66c2e7100e5b9354ac5a6e454e
Sep 07 10:36:03.659255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:03.659218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx" event={"ID":"4c5d87ec-03d1-4f85-98c1-9389de56dc9d","Type":"ContainerStarted","Data":"018272e945d4d46ade9152c313b6e93b32c04a7be1b36f150393f37ab5ec22f8"}
Sep 07 10:36:03.659639 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:03.659263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx" event={"ID":"4c5d87ec-03d1-4f85-98c1-9389de56dc9d","Type":"ContainerStarted","Data":"cb94ddd66816a200f6e13ac6ddfdc2d32bd89a66c2e7100e5b9354ac5a6e454e"}
Sep 07 10:36:03.695783 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:03.695736    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx" podStartSLOduration=1.695722257 podStartE2EDuration="1.695722257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:36:03.692319658 +0000 UTC m=+1470.240834645" watchObservedRunningTime="2026-09-07 10:36:03.695722257 +0000 UTC m=+1470.244237274"
Sep 07 10:36:09.929888 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:09.929850    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:10.029908 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:10.029846    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:10.151100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:10.151040    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:10.274964 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:10.274873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:10.394953 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:10.394918    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:10.540163 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:10.540125    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:36:10.735234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:10.735192    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:12.820351 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:12.820310    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:12.820768 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:12.820584    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:12.832974 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:12.832944    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:13.717520 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:13.717489    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:36:19.119652 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:19.119612    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:19.901211 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:19.901170    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:22.419140 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:22.419108    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:36:26.496085 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:26.496037    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:29.955494 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:29.955445    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:29.982661 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:29.982623    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"]
Sep 07 10:36:29.982924 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:29.982893    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz" podUID="e6ff1335-e8ff-460e-ae49-cb2f3b160df1" containerName="payload-pre-processing" containerID="cri-o://afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744" gracePeriod=30
Sep 07 10:36:30.105188 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.105153    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:30.233498 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.233473    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:36:30.284178 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.284141    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:30.299999 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.299970    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvgxp\" (UniqueName: \"kubernetes.io/projected/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-kube-api-access-rvgxp\") pod \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\" (UID: \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\") "
Sep 07 10:36:30.300148 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.300042    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-plugins-config-volume\") pod \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\" (UID: \"e6ff1335-e8ff-460e-ae49-cb2f3b160df1\") "
Sep 07 10:36:30.300452 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.300426    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6ff1335-e8ff-460e-ae49-cb2f3b160df1" (UID: "e6ff1335-e8ff-460e-ae49-cb2f3b160df1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:30.302133 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.302116    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-kube-api-access-rvgxp" (OuterVolumeSpecName: "kube-api-access-rvgxp") pod "e6ff1335-e8ff-460e-ae49-cb2f3b160df1" (UID: "e6ff1335-e8ff-460e-ae49-cb2f3b160df1"). InnerVolumeSpecName "kube-api-access-rvgxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:30.382243 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.382205    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:30.401501 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.401478    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvgxp\" (UniqueName: \"kubernetes.io/projected/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-kube-api-access-rvgxp\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:36:30.401501 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.401499    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ff1335-e8ff-460e-ae49-cb2f3b160df1-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:36:30.707684 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.707646    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:30.792879 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.792847    2560 generic.go:356] "Generic (PLEG): container finished" podID="e6ff1335-e8ff-460e-ae49-cb2f3b160df1" containerID="afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744" exitCode=0
Sep 07 10:36:30.793033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.792885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz" event={"ID":"e6ff1335-e8ff-460e-ae49-cb2f3b160df1","Type":"ContainerDied","Data":"afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744"}
Sep 07 10:36:30.793033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.792894    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"
Sep 07 10:36:30.793033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.792925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz" event={"ID":"e6ff1335-e8ff-460e-ae49-cb2f3b160df1","Type":"ContainerDied","Data":"718072911f7412bf77e9e7c24f549315e1547f288709b9c7213db23da2b55d93"}
Sep 07 10:36:30.793033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.792947    2560 scope.go:117] "RemoveContainer" containerID="afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744"
Sep 07 10:36:30.801746 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.801722    2560 scope.go:117] "RemoveContainer" containerID="afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744"
Sep 07 10:36:30.802103 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:36:30.802036    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744\": container with ID starting with afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744 not found: ID does not exist" containerID="afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744"
Sep 07 10:36:30.802169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.802115    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744"} err="failed to get container status \"afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744\": rpc error: code = NotFound desc = could not find container \"afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744\": container with ID starting with afdc34ee941ea1fdb573450d689583396482b0b043f86eff113eba6d61a51744 not found: ID does not exist"
Sep 07 10:36:30.839673 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.839638    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"]
Sep 07 10:36:30.841346 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:30.841327    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bdcc7-855b5c5c94-bxtvz"]
Sep 07 10:36:31.895621 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:31.895586    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6ff1335-e8ff-460e-ae49-cb2f3b160df1" path="/var/lib/kubelet/pods/e6ff1335-e8ff-460e-ae49-cb2f3b160df1/volumes"
Sep 07 10:36:34.872409 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:34.872372    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:36:35.509810 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:35.509771    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:38.300503 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:38.300466    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:38.408340 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:38.408290    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:38.703048 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:38.702944    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:39.112453 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:39.112418    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:49.851056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:49.851021    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"]
Sep 07 10:36:49.853455 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:49.851237    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q" podUID="a4323079-7fdb-4156-92cb-61af62f109dd" containerName="maas-api" containerID="cri-o://deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81" gracePeriod=30
Sep 07 10:36:49.916965 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:49.916932    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:50.000553 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:49.998184    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"]
Sep 07 10:36:50.000553 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:49.998506    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s" podUID="281309a8-e027-425b-b375-4179545fc516" containerName="payload-processing" containerID="cri-o://81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3" gracePeriod=30
Sep 07 10:36:50.119351 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.119327    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:36:50.156167 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.156131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:50.251836 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.251808    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:36:50.280804 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.280781    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-metrics-certs\") pod \"a4323079-7fdb-4156-92cb-61af62f109dd\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") "
Sep 07 10:36:50.280885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.280809    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4323079-7fdb-4156-92cb-61af62f109dd-openshift-service-ca\") pod \"a4323079-7fdb-4156-92cb-61af62f109dd\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") "
Sep 07 10:36:50.280885 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.280838    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls\") pod \"a4323079-7fdb-4156-92cb-61af62f109dd\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") "
Sep 07 10:36:50.281007 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.280903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkshk\" (UniqueName: \"kubernetes.io/projected/a4323079-7fdb-4156-92cb-61af62f109dd-kube-api-access-rkshk\") pod \"a4323079-7fdb-4156-92cb-61af62f109dd\" (UID: \"a4323079-7fdb-4156-92cb-61af62f109dd\") "
Sep 07 10:36:50.281193 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.281170    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4323079-7fdb-4156-92cb-61af62f109dd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4323079-7fdb-4156-92cb-61af62f109dd" (UID: "a4323079-7fdb-4156-92cb-61af62f109dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:50.282988 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.282958    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4323079-7fdb-4156-92cb-61af62f109dd" (UID: "a4323079-7fdb-4156-92cb-61af62f109dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:36:50.283100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.283009    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4323079-7fdb-4156-92cb-61af62f109dd-kube-api-access-rkshk" (OuterVolumeSpecName: "kube-api-access-rkshk") pod "a4323079-7fdb-4156-92cb-61af62f109dd" (UID: "a4323079-7fdb-4156-92cb-61af62f109dd"). InnerVolumeSpecName "kube-api-access-rkshk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:50.283100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.283027    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4323079-7fdb-4156-92cb-61af62f109dd" (UID: "a4323079-7fdb-4156-92cb-61af62f109dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:36:50.382306 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.382252    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281309a8-e027-425b-b375-4179545fc516-plugins-config-volume\") pod \"281309a8-e027-425b-b375-4179545fc516\" (UID: \"281309a8-e027-425b-b375-4179545fc516\") "
Sep 07 10:36:50.382306 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.382303    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ldrf\" (UniqueName: \"kubernetes.io/projected/281309a8-e027-425b-b375-4179545fc516-kube-api-access-2ldrf\") pod \"281309a8-e027-425b-b375-4179545fc516\" (UID: \"281309a8-e027-425b-b375-4179545fc516\") "
Sep 07 10:36:50.382507 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.382495    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-metrics-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:36:50.382576 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.382508    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4323079-7fdb-4156-92cb-61af62f109dd-openshift-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:36:50.382576 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.382517    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4323079-7fdb-4156-92cb-61af62f109dd-maas-api-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:36:50.382576 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.382525    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkshk\" (UniqueName: \"kubernetes.io/projected/a4323079-7fdb-4156-92cb-61af62f109dd-kube-api-access-rkshk\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:36:50.382576 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.382548    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281309a8-e027-425b-b375-4179545fc516-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "281309a8-e027-425b-b375-4179545fc516" (UID: "281309a8-e027-425b-b375-4179545fc516"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:36:50.384137 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.384118    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281309a8-e027-425b-b375-4179545fc516-kube-api-access-2ldrf" (OuterVolumeSpecName: "kube-api-access-2ldrf") pod "281309a8-e027-425b-b375-4179545fc516" (UID: "281309a8-e027-425b-b375-4179545fc516"). InnerVolumeSpecName "kube-api-access-2ldrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:36:50.415396 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.415368    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:50.483288 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.483265    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281309a8-e027-425b-b375-4179545fc516-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:36:50.483288 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.483286    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldrf\" (UniqueName: \"kubernetes.io/projected/281309a8-e027-425b-b375-4179545fc516-kube-api-access-2ldrf\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:36:50.624014 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.623987    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:50.888410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.888384    2560 generic.go:356] "Generic (PLEG): container finished" podID="281309a8-e027-425b-b375-4179545fc516" containerID="81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3" exitCode=0
Sep 07 10:36:50.888714 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.888423    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"
Sep 07 10:36:50.888714 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.888472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s" event={"ID":"281309a8-e027-425b-b375-4179545fc516","Type":"ContainerDied","Data":"81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3"}
Sep 07 10:36:50.888714 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.888521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s" event={"ID":"281309a8-e027-425b-b375-4179545fc516","Type":"ContainerDied","Data":"764d99b94c7354303b195c149be45a293e04f615806686502fc65693b30758f9"}
Sep 07 10:36:50.888714 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.888550    2560 scope.go:117] "RemoveContainer" containerID="81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3"
Sep 07 10:36:50.889969 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.889950    2560 generic.go:356] "Generic (PLEG): container finished" podID="a4323079-7fdb-4156-92cb-61af62f109dd" containerID="deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81" exitCode=0
Sep 07 10:36:50.890130 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.889984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q" event={"ID":"a4323079-7fdb-4156-92cb-61af62f109dd","Type":"ContainerDied","Data":"deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81"}
Sep 07 10:36:50.890130 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.890001    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"
Sep 07 10:36:50.890130 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.890017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q" event={"ID":"a4323079-7fdb-4156-92cb-61af62f109dd","Type":"ContainerDied","Data":"6f0b92f6581dd25a67b5df5ca57d894bda4f1deaaaf10b0db2f6e931604d7029"}
Sep 07 10:36:50.898439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.898423    2560 scope.go:117] "RemoveContainer" containerID="81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3"
Sep 07 10:36:50.898737 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:36:50.898717    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3\": container with ID starting with 81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3 not found: ID does not exist" containerID="81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3"
Sep 07 10:36:50.898844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.898740    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3"} err="failed to get container status \"81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3\": rpc error: code = NotFound desc = could not find container \"81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3\": container with ID starting with 81ae2aa76a19bd7896d9413f673d04a9f22783b67a0d006b92a79a2ccd7b23a3 not found: ID does not exist"
Sep 07 10:36:50.898844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.898756    2560 scope.go:117] "RemoveContainer" containerID="deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81"
Sep 07 10:36:50.906828 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.906813    2560 scope.go:117] "RemoveContainer" containerID="deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81"
Sep 07 10:36:50.907093 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:36:50.907057    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81\": container with ID starting with deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81 not found: ID does not exist" containerID="deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81"
Sep 07 10:36:50.907151 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.907099    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81"} err="failed to get container status \"deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81\": rpc error: code = NotFound desc = could not find container \"deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81\": container with ID starting with deb7034ba88aa028f932dfe997ca4269b61a2d9ba507329e66ae56b7d3a22e81 not found: ID does not exist"
Sep 07 10:36:50.920908 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.920889    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"]
Sep 07 10:36:50.927483 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.927463    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-319d4c-68c99449b7-sth4s"]
Sep 07 10:36:50.954544 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.954526    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"]
Sep 07 10:36:50.961014 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.960994    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-319d4c-b8f947647-5nm7q"]
Sep 07 10:36:50.969972 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:50.969947    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:51.896800 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:51.896767    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="281309a8-e027-425b-b375-4179545fc516" path="/var/lib/kubelet/pods/281309a8-e027-425b-b375-4179545fc516/volumes"
Sep 07 10:36:51.897298 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:51.897275    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4323079-7fdb-4156-92cb-61af62f109dd" path="/var/lib/kubelet/pods/a4323079-7fdb-4156-92cb-61af62f109dd/volumes"
Sep 07 10:36:55.850011 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:55.849973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:36:58.577468 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:58.577376    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:58.735681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:58.735651    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:36:58.969689 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:36:58.969603    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:09.427133 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:09.427102    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:37:19.512795 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:19.512754    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:37:24.907926 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:24.907892    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"]
Sep 07 10:37:24.951385 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:24.951354    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:25.976136 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:25.976085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:26.415276 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:26.415237    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:26.946463 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:26.946429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:27.122038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:27.122006    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:31.062731 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:31.062692    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:31.219939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:31.219906    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:32.501484 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:32.501447    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:33.827056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827027    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"]
Sep 07 10:37:33.827445 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827379    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6ff1335-e8ff-460e-ae49-cb2f3b160df1" containerName="payload-pre-processing"
Sep 07 10:37:33.827445 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827391    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6ff1335-e8ff-460e-ae49-cb2f3b160df1" containerName="payload-pre-processing"
Sep 07 10:37:33.827445 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827400    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281309a8-e027-425b-b375-4179545fc516" containerName="payload-processing"
Sep 07 10:37:33.827445 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827406    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="281309a8-e027-425b-b375-4179545fc516" containerName="payload-processing"
Sep 07 10:37:33.827445 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827422    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4323079-7fdb-4156-92cb-61af62f109dd" containerName="maas-api"
Sep 07 10:37:33.827445 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827427    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4323079-7fdb-4156-92cb-61af62f109dd" containerName="maas-api"
Sep 07 10:37:33.827629 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827485    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6ff1335-e8ff-460e-ae49-cb2f3b160df1" containerName="payload-pre-processing"
Sep 07 10:37:33.827629 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827492    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="281309a8-e027-425b-b375-4179545fc516" containerName="payload-processing"
Sep 07 10:37:33.827629 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.827498    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4323079-7fdb-4156-92cb-61af62f109dd" containerName="maas-api"
Sep 07 10:37:33.830785 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.830766    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:33.837974 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:37:33.837948    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47\" is forbidden: User \"system:node:ip-10-0-128-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-191.ec2.internal' and this object" podUID="0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:33.838093 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:37:33.837976    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-ipp-a-92aa2e-dockercfg-klbqj\" is forbidden: User \"system:node:ip-10-0-128-191.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-191.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-92aa2e-dockercfg-klbqj\"" type="*v1.Secret"
Sep 07 10:37:33.838093 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:37:33.837985    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-ipp-a-92aa2e\" is forbidden: User \"system:node:ip-10-0-128-191.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-191.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-92aa2e\"" type="*v1.ConfigMap"
Sep 07 10:37:33.858499 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.858479    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"]
Sep 07 10:37:33.895252 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.895233    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"]
Sep 07 10:37:33.898368 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.898354    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:33.929900 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.929876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw78c\" (UniqueName: \"kubernetes.io/projected/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-kube-api-access-hw78c\") pod \"payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47\" (UID: \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:33.930089 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.930049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnc7\" (UniqueName: \"kubernetes.io/projected/82931fe9-8e06-4bf9-aa53-b9de440bcba9-kube-api-access-hdnc7\") pod \"payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6\" (UID: \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:33.930254 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.930216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82931fe9-8e06-4bf9-aa53-b9de440bcba9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6\" (UID: \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:33.930365 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.930332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47\" (UID: \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:33.931521 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:33.931502    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"]
Sep 07 10:37:34.025511 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.025489    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:34.030992 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.030972    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47\" (UID: \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:34.031130 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.031012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw78c\" (UniqueName: \"kubernetes.io/projected/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-kube-api-access-hw78c\") pod \"payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47\" (UID: \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:34.031130 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.031041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnc7\" (UniqueName: \"kubernetes.io/projected/82931fe9-8e06-4bf9-aa53-b9de440bcba9-kube-api-access-hdnc7\") pod \"payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6\" (UID: \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:34.031130 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.031098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82931fe9-8e06-4bf9-aa53-b9de440bcba9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6\" (UID: \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:34.044982 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.044957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw78c\" (UniqueName: \"kubernetes.io/projected/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-kube-api-access-hw78c\") pod \"payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47\" (UID: \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:34.045237 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.045215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnc7\" (UniqueName: \"kubernetes.io/projected/82931fe9-8e06-4bf9-aa53-b9de440bcba9-kube-api-access-hdnc7\") pod \"payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6\" (UID: \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:34.895254 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.895223    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-92aa2e\""
Sep 07 10:37:34.901876 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.901855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47\" (UID: \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:34.901953 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:34.901854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82931fe9-8e06-4bf9-aa53-b9de440bcba9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6\" (UID: \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:35.422260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:35.422227    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-92aa2e-dockercfg-klbqj\""
Sep 07 10:37:35.427673 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:35.427653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:35.430438 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:35.430418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:35.728546 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:35.728477    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"]
Sep 07 10:37:35.731793 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:37:35.731766    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82931fe9_8e06_4bf9_aa53_b9de440bcba9.slice/crio-3ae3772ed97bdf987dd8109a66d3fa9f54f3053b8870f09bf8816763f372c76f WatchSource:0}: Error finding container 3ae3772ed97bdf987dd8109a66d3fa9f54f3053b8870f09bf8816763f372c76f: Status 404 returned error can't find the container with id 3ae3772ed97bdf987dd8109a66d3fa9f54f3053b8870f09bf8816763f372c76f
Sep 07 10:37:35.757026 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:35.757004    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:37:35.772295 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:35.772254    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"]
Sep 07 10:37:35.774027 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:37:35.774003    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0685cd14_5ed0_4791_b6d8_e0272a8d2e6c.slice/crio-244113e99970cd4798e687c2c3eadf1a023d9cef75e1bf4f058f0fadf7bc1449 WatchSource:0}: Error finding container 244113e99970cd4798e687c2c3eadf1a023d9cef75e1bf4f058f0fadf7bc1449: Status 404 returned error can't find the container with id 244113e99970cd4798e687c2c3eadf1a023d9cef75e1bf4f058f0fadf7bc1449
Sep 07 10:37:36.097847 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:36.097807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6" event={"ID":"82931fe9-8e06-4bf9-aa53-b9de440bcba9","Type":"ContainerStarted","Data":"56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d"}
Sep 07 10:37:36.098312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:36.097870    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:36.098312 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:36.097896    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6" event={"ID":"82931fe9-8e06-4bf9-aa53-b9de440bcba9","Type":"ContainerStarted","Data":"3ae3772ed97bdf987dd8109a66d3fa9f54f3053b8870f09bf8816763f372c76f"}
Sep 07 10:37:36.099254 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:36.099235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47" event={"ID":"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c","Type":"ContainerStarted","Data":"74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18"}
Sep 07 10:37:36.099322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:36.099270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47" event={"ID":"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c","Type":"ContainerStarted","Data":"244113e99970cd4798e687c2c3eadf1a023d9cef75e1bf4f058f0fadf7bc1449"}
Sep 07 10:37:36.099322 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:36.099290    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:36.139676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:36.139635    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6" podStartSLOduration=3.139620716 podStartE2EDuration="3.139620716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:37:36.138635662 +0000 UTC m=+1562.687150647" watchObservedRunningTime="2026-09-07 10:37:36.139620716 +0000 UTC m=+1562.688135702"
Sep 07 10:37:36.173297 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:36.173250    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47" podStartSLOduration=3.173232733 podStartE2EDuration="3.173232733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:37:36.170849204 +0000 UTC m=+1562.719364191" watchObservedRunningTime="2026-09-07 10:37:36.173232733 +0000 UTC m=+1562.721747723"
Sep 07 10:37:39.926600 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:39.926556    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx" podUID="4c5d87ec-03d1-4f85-98c1-9389de56dc9d" containerName="main" containerID="cri-o://018272e945d4d46ade9152c313b6e93b32c04a7be1b36f150393f37ab5ec22f8" gracePeriod=45
Sep 07 10:37:40.120298 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.120271    2560 generic.go:356] "Generic (PLEG): container finished" podID="4c5d87ec-03d1-4f85-98c1-9389de56dc9d" containerID="018272e945d4d46ade9152c313b6e93b32c04a7be1b36f150393f37ab5ec22f8" exitCode=0
Sep 07 10:37:40.120458 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.120357    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx" event={"ID":"4c5d87ec-03d1-4f85-98c1-9389de56dc9d","Type":"ContainerDied","Data":"018272e945d4d46ade9152c313b6e93b32c04a7be1b36f150393f37ab5ec22f8"}
Sep 07 10:37:40.173614 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.173578    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:37:40.286169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.286142    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2t27\" (UniqueName: \"kubernetes.io/projected/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-kube-api-access-x2t27\") pod \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") "
Sep 07 10:37:40.286310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.286200    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-model-cache\") pod \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") "
Sep 07 10:37:40.286310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.286233    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tmp-dir\") pod \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") "
Sep 07 10:37:40.286310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.286264    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tls-certs\") pod \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") "
Sep 07 10:37:40.286310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.286283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-dshm\") pod \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") "
Sep 07 10:37:40.286522 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.286428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-home\") pod \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\" (UID: \"4c5d87ec-03d1-4f85-98c1-9389de56dc9d\") "
Sep 07 10:37:40.286522 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.286505    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c5d87ec-03d1-4f85-98c1-9389de56dc9d" (UID: "4c5d87ec-03d1-4f85-98c1-9389de56dc9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:37:40.286614 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.286517    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c5d87ec-03d1-4f85-98c1-9389de56dc9d" (UID: "4c5d87ec-03d1-4f85-98c1-9389de56dc9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:37:40.288164 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.287365    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-home" (OuterVolumeSpecName: "home") pod "4c5d87ec-03d1-4f85-98c1-9389de56dc9d" (UID: "4c5d87ec-03d1-4f85-98c1-9389de56dc9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:37:40.288164 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.287720    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-model-cache\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:37:40.288164 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.287743    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tmp-dir\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:37:40.289116 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.289084    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c5d87ec-03d1-4f85-98c1-9389de56dc9d" (UID: "4c5d87ec-03d1-4f85-98c1-9389de56dc9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:37:40.294549 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.294312    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-dshm" (OuterVolumeSpecName: "dshm") pod "4c5d87ec-03d1-4f85-98c1-9389de56dc9d" (UID: "4c5d87ec-03d1-4f85-98c1-9389de56dc9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:37:40.294678 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.294629    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-kube-api-access-x2t27" (OuterVolumeSpecName: "kube-api-access-x2t27") pod "4c5d87ec-03d1-4f85-98c1-9389de56dc9d" (UID: "4c5d87ec-03d1-4f85-98c1-9389de56dc9d"). InnerVolumeSpecName "kube-api-access-x2t27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:37:40.388155 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.388133    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-home\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:37:40.388255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.388156    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2t27\" (UniqueName: \"kubernetes.io/projected/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-kube-api-access-x2t27\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:37:40.388255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.388168    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-tls-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:37:40.388255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:40.388176    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c5d87ec-03d1-4f85-98c1-9389de56dc9d-dshm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:37:41.126244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:41.126215    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"
Sep 07 10:37:41.126244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:41.126235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx" event={"ID":"4c5d87ec-03d1-4f85-98c1-9389de56dc9d","Type":"ContainerDied","Data":"cb94ddd66816a200f6e13ac6ddfdc2d32bd89a66c2e7100e5b9354ac5a6e454e"}
Sep 07 10:37:41.127092 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:41.126285    2560 scope.go:117] "RemoveContainer" containerID="018272e945d4d46ade9152c313b6e93b32c04a7be1b36f150393f37ab5ec22f8"
Sep 07 10:37:41.155255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:41.155230    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"]
Sep 07 10:37:41.159381 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:41.159361    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-0a577c/rate-test-model-0a577c-kserve-db7785679-zl5mx"]
Sep 07 10:37:41.894896 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:41.894866    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c5d87ec-03d1-4f85-98c1-9389de56dc9d" path="/var/lib/kubelet/pods/4c5d87ec-03d1-4f85-98c1-9389de56dc9d/volumes"
Sep 07 10:37:47.107848 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:47.107809    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:37:47.108287 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:47.107995    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:37:56.003507 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:56.003467    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:56.644135 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:56.644085    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:57.004914 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:57.004837    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:57.185707 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:57.185680    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:59.750582 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:59.750541    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:37:59.840154 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:37:59.840119    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:00.926745 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:00.926704    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:01.027844 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.027806    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:01.350426 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.350395    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:01.509699 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.509675    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:01.608998 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.608914    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:01.692292 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.692266    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"]
Sep 07 10:38:01.692649 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.692637    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c5d87ec-03d1-4f85-98c1-9389de56dc9d" containerName="main"
Sep 07 10:38:01.692693 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.692651    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c5d87ec-03d1-4f85-98c1-9389de56dc9d" containerName="main"
Sep 07 10:38:01.692734 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.692715    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c5d87ec-03d1-4f85-98c1-9389de56dc9d" containerName="main"
Sep 07 10:38:01.696995 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.696979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:01.699847 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.699829    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-125f28\""
Sep 07 10:38:01.699958 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.699885    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-125f28-dockercfg-54kh7\""
Sep 07 10:38:01.707121 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.707100    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"]
Sep 07 10:38:01.876846 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.876780    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm\" (UID: \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:01.876846 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.876838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz864\" (UniqueName: \"kubernetes.io/projected/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-kube-api-access-qz864\") pod \"payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm\" (UID: \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:01.977281 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.977256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm\" (UID: \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:01.977595 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.977313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz864\" (UniqueName: \"kubernetes.io/projected/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-kube-api-access-qz864\") pod \"payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm\" (UID: \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:01.977921 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.977901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm\" (UID: \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:01.986549 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:01.986525    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz864\" (UniqueName: \"kubernetes.io/projected/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-kube-api-access-qz864\") pod \"payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm\" (UID: \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:02.007956 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:02.007932    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:02.276853 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:02.276766    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"]
Sep 07 10:38:02.279623 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:38:02.279597    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3d95c0_fad7_4a54_9604_c0cde9f2ba19.slice/crio-da86f481069d8b570291d1053100b4b0e321de1a38a7f4c506744a61e1805d6d WatchSource:0}: Error finding container da86f481069d8b570291d1053100b4b0e321de1a38a7f4c506744a61e1805d6d: Status 404 returned error can't find the container with id da86f481069d8b570291d1053100b4b0e321de1a38a7f4c506744a61e1805d6d
Sep 07 10:38:03.232444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:03.232412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm" event={"ID":"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19","Type":"ContainerStarted","Data":"6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97"}
Sep 07 10:38:03.232444 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:03.232459    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm" event={"ID":"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19","Type":"ContainerStarted","Data":"da86f481069d8b570291d1053100b4b0e321de1a38a7f4c506744a61e1805d6d"}
Sep 07 10:38:03.232932 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:03.232502    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:03.250616 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:03.250572    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm" podStartSLOduration=2.250559105 podStartE2EDuration="2.250559105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:03.249414559 +0000 UTC m=+1589.797929561" watchObservedRunningTime="2026-09-07 10:38:03.250559105 +0000 UTC m=+1589.799074123"
Sep 07 10:38:05.988183 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:05.988150    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:08.216957 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:08.216918    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:38:14.238943 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:14.238915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:38:24.683029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:24.682991    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:43.213282 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.213248    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"]
Sep 07 10:38:43.216797 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.216776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.220423 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.220381    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-92aa2e\"/\"default-dockercfg-r974j\""
Sep 07 10:38:43.220423 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.220427    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-92aa2e\"/\"openshift-service-ca.crt\""
Sep 07 10:38:43.220624 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.220451    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-92aa2e\"/\"ipp-route-92aa2e-kserve-self-signed-certs\""
Sep 07 10:38:43.220624 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.220552    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-92aa2e\"/\"kube-root-ca.crt\""
Sep 07 10:38:43.226729 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.226708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"]
Sep 07 10:38:43.250110 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.250061    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:43.315540 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.315490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-home\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.315758 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.315547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f65a758-ea00-41be-a602-9b7ab044d7c5-tls-certs\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.315758 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.315642    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-tmp-dir\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.315758 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.315710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-dshm\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.315758 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.315744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdlr\" (UniqueName: \"kubernetes.io/projected/3f65a758-ea00-41be-a602-9b7ab044d7c5-kube-api-access-bjdlr\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.316217 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.316177    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-model-cache\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.417303 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.417267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-model-cache\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.417303 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.417308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-home\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.417566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.417332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f65a758-ea00-41be-a602-9b7ab044d7c5-tls-certs\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.417566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.417358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-tmp-dir\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.417680 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.417659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-dshm\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.417733 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.417701    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdlr\" (UniqueName: \"kubernetes.io/projected/3f65a758-ea00-41be-a602-9b7ab044d7c5-kube-api-access-bjdlr\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.417786 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.417743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-home\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.417786 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.417756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-tmp-dir\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.418102 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.418062    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-model-cache\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.419912 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.419887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-dshm\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.420563 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.420541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f65a758-ea00-41be-a602-9b7ab044d7c5-tls-certs\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.431333 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.431310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdlr\" (UniqueName: \"kubernetes.io/projected/3f65a758-ea00-41be-a602-9b7ab044d7c5-kube-api-access-bjdlr\") pod \"ipp-route-92aa2e-kserve-688db8964c-cnq4n\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") " pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.526707 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.526603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:43.799841 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:43.799815    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"]
Sep 07 10:38:43.801663 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:38:43.801636    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f65a758_ea00_41be_a602_9b7ab044d7c5.slice/crio-0a77f55a09227e896f7a5a0a98c01cf03014925897e18f02c80fc49b550358bf WatchSource:0}: Error finding container 0a77f55a09227e896f7a5a0a98c01cf03014925897e18f02c80fc49b550358bf: Status 404 returned error can't find the container with id 0a77f55a09227e896f7a5a0a98c01cf03014925897e18f02c80fc49b550358bf
Sep 07 10:38:44.426046 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:44.426010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n" event={"ID":"3f65a758-ea00-41be-a602-9b7ab044d7c5","Type":"ContainerStarted","Data":"dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340"}
Sep 07 10:38:44.426046 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:44.426052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n" event={"ID":"3f65a758-ea00-41be-a602-9b7ab044d7c5","Type":"ContainerStarted","Data":"0a77f55a09227e896f7a5a0a98c01cf03014925897e18f02c80fc49b550358bf"}
Sep 07 10:38:44.450148 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:44.450092    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n" podStartSLOduration=1.450064094 podStartE2EDuration="1.450064094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:38:44.446942057 +0000 UTC m=+1630.995457042" watchObservedRunningTime="2026-09-07 10:38:44.450064094 +0000 UTC m=+1630.998579080"
Sep 07 10:38:48.749641 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:48.749607    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:38:48.808566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:48.808534    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:38:53.527343 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:53.527302    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:53.527741 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:53.527414    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:53.539721 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:53.539699    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:54.487737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:54.487704    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:38:59.902242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:38:59.902208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:39:05.355523 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:05.355488    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6_82931fe9-8e06-4bf9-aa53-b9de440bcba9/payload-processing/0.log"
Sep 07 10:39:05.847163 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:05.847132    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:06.204825 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:06.204752    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:06.283576 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:06.283541    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:09.104414 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:09.104353    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:09.202335 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:09.202291    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:10.452016 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:10.451971    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:10.558321 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:10.558284    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:10.852056 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:10.852027    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:11.107260 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.107179    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:11.207517 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.207476    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:11.252341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.252312    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"]
Sep 07 10:39:11.256035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.256012    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:11.258950 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.258935    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c43305-dockercfg-9qk7n\""
Sep 07 10:39:11.259257 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.259242    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c43305\""
Sep 07 10:39:11.284228 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.284205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"]
Sep 07 10:39:11.364621 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.364565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac78cbd1-4239-451f-8844-c80ff87b8f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq\" (UID: \"ac78cbd1-4239-451f-8844-c80ff87b8f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:11.364714 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.364673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l22q\" (UniqueName: \"kubernetes.io/projected/ac78cbd1-4239-451f-8844-c80ff87b8f20-kube-api-access-8l22q\") pod \"payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq\" (UID: \"ac78cbd1-4239-451f-8844-c80ff87b8f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:11.465373 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.465338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l22q\" (UniqueName: \"kubernetes.io/projected/ac78cbd1-4239-451f-8844-c80ff87b8f20-kube-api-access-8l22q\") pod \"payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq\" (UID: \"ac78cbd1-4239-451f-8844-c80ff87b8f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:11.465716 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.465388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac78cbd1-4239-451f-8844-c80ff87b8f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq\" (UID: \"ac78cbd1-4239-451f-8844-c80ff87b8f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:11.466015 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.465996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac78cbd1-4239-451f-8844-c80ff87b8f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq\" (UID: \"ac78cbd1-4239-451f-8844-c80ff87b8f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:11.482264 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.482238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l22q\" (UniqueName: \"kubernetes.io/projected/ac78cbd1-4239-451f-8844-c80ff87b8f20-kube-api-access-8l22q\") pod \"payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq\" (UID: \"ac78cbd1-4239-451f-8844-c80ff87b8f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:11.566202 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.566178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:11.836192 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:11.836169    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"]
Sep 07 10:39:11.838324 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:39:11.838298    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac78cbd1_4239_451f_8844_c80ff87b8f20.slice/crio-514f844254ccf8537fc79306828d8be3fae269c051491a00588ecd4484cf22ff WatchSource:0}: Error finding container 514f844254ccf8537fc79306828d8be3fae269c051491a00588ecd4484cf22ff: Status 404 returned error can't find the container with id 514f844254ccf8537fc79306828d8be3fae269c051491a00588ecd4484cf22ff
Sep 07 10:39:12.566679 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:12.566643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq" event={"ID":"ac78cbd1-4239-451f-8844-c80ff87b8f20","Type":"ContainerStarted","Data":"ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e"}
Sep 07 10:39:12.567178 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:12.566699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq" event={"ID":"ac78cbd1-4239-451f-8844-c80ff87b8f20","Type":"ContainerStarted","Data":"514f844254ccf8537fc79306828d8be3fae269c051491a00588ecd4484cf22ff"}
Sep 07 10:39:12.567178 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:12.566781    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:12.586247 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:12.586198    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq" podStartSLOduration=1.586180566 podStartE2EDuration="1.586180566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:39:12.58385441 +0000 UTC m=+1659.132369397" watchObservedRunningTime="2026-09-07 10:39:12.586180566 +0000 UTC m=+1659.134695552"
Sep 07 10:39:18.100160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:18.100130    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:39:19.618416 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:19.618358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:23.574259 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:23.574221    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:31.702577 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:31.702540    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:31.757118 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:31.757087    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:31.834836 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:31.834806    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:34.076419 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:34.076358    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:40.923735 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:40.923685    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:40.954420 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:40.954353    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"]
Sep 07 10:39:40.954620 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:40.954592    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq" podUID="ac78cbd1-4239-451f-8844-c80ff87b8f20" containerName="payload-processing" containerID="cri-o://ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e" gracePeriod=30
Sep 07 10:39:41.027291 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.027257    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:41.125448 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.125415    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:41.199660 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.199635    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:41.224840 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.224800    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:41.316018 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.315996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac78cbd1-4239-451f-8844-c80ff87b8f20-plugins-config-volume\") pod \"ac78cbd1-4239-451f-8844-c80ff87b8f20\" (UID: \"ac78cbd1-4239-451f-8844-c80ff87b8f20\") "
Sep 07 10:39:41.316146 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.316049    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l22q\" (UniqueName: \"kubernetes.io/projected/ac78cbd1-4239-451f-8844-c80ff87b8f20-kube-api-access-8l22q\") pod \"ac78cbd1-4239-451f-8844-c80ff87b8f20\" (UID: \"ac78cbd1-4239-451f-8844-c80ff87b8f20\") "
Sep 07 10:39:41.316394 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.316354    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac78cbd1-4239-451f-8844-c80ff87b8f20-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac78cbd1-4239-451f-8844-c80ff87b8f20" (UID: "ac78cbd1-4239-451f-8844-c80ff87b8f20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:39:41.318013 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.317988    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac78cbd1-4239-451f-8844-c80ff87b8f20-kube-api-access-8l22q" (OuterVolumeSpecName: "kube-api-access-8l22q") pod "ac78cbd1-4239-451f-8844-c80ff87b8f20" (UID: "ac78cbd1-4239-451f-8844-c80ff87b8f20"). InnerVolumeSpecName "kube-api-access-8l22q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:39:41.417100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.417057    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l22q\" (UniqueName: \"kubernetes.io/projected/ac78cbd1-4239-451f-8844-c80ff87b8f20-kube-api-access-8l22q\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:39:41.417100 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.417099    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac78cbd1-4239-451f-8844-c80ff87b8f20-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:39:41.510361 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.510278    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:41.714920 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.714890    2560 generic.go:356] "Generic (PLEG): container finished" podID="ac78cbd1-4239-451f-8844-c80ff87b8f20" containerID="ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e" exitCode=0
Sep 07 10:39:41.715064 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.714943    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"
Sep 07 10:39:41.715064 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.714977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq" event={"ID":"ac78cbd1-4239-451f-8844-c80ff87b8f20","Type":"ContainerDied","Data":"ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e"}
Sep 07 10:39:41.715064 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.715046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq" event={"ID":"ac78cbd1-4239-451f-8844-c80ff87b8f20","Type":"ContainerDied","Data":"514f844254ccf8537fc79306828d8be3fae269c051491a00588ecd4484cf22ff"}
Sep 07 10:39:41.715213 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.715087    2560 scope.go:117] "RemoveContainer" containerID="ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e"
Sep 07 10:39:41.724433 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.724418    2560 scope.go:117] "RemoveContainer" containerID="ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e"
Sep 07 10:39:41.724691 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:39:41.724671    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e\": container with ID starting with ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e not found: ID does not exist" containerID="ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e"
Sep 07 10:39:41.724754 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.724698    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e"} err="failed to get container status \"ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e\": rpc error: code = NotFound desc = could not find container \"ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e\": container with ID starting with ce4f20680972244551b751bf04dfbacd6f5786060fca8523075274ad0b24161e not found: ID does not exist"
Sep 07 10:39:41.738116 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.738094    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"]
Sep 07 10:39:41.741667 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.741642    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c43305-55f757d595-s9qwq"]
Sep 07 10:39:41.895677 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:41.895652    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac78cbd1-4239-451f-8844-c80ff87b8f20" path="/var/lib/kubelet/pods/ac78cbd1-4239-451f-8844-c80ff87b8f20/volumes"
Sep 07 10:39:47.194359 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:47.194326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:39:47.584597 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:47.584556    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:47.685798 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:47.685763    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:47.915699 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:47.915616    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:48.137168 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:48.137129    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:39:51.530479 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:39:51.530438    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:05.901799 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:05.901722    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:40:06.930487 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:06.930455    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:06.949856 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:06.949821    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"]
Sep 07 10:40:06.950117 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:06.950090    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6" podUID="82931fe9-8e06-4bf9-aa53-b9de440bcba9" containerName="payload-processing" containerID="cri-o://56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d" gracePeriod=30
Sep 07 10:40:06.957273 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:06.957247    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"]
Sep 07 10:40:06.958014 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:06.957500    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47" podUID="0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" containerName="payload-pre-processing" containerID="cri-o://74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18" gracePeriod=30
Sep 07 10:40:07.030608 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.030576    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:07.106173 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.106127    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47" podUID="0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.64:9004: connect: connection refused"
Sep 07 10:40:07.106347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.106131    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6" podUID="82931fe9-8e06-4bf9-aa53-b9de440bcba9" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.65:9004: connect: connection refused"
Sep 07 10:40:07.128561 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.128522    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:07.201583 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.201551    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:07.222875 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.222852    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:40:07.225653 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.225632    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:40:07.334254 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.334226    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw78c\" (UniqueName: \"kubernetes.io/projected/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-kube-api-access-hw78c\") pod \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\" (UID: \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\") "
Sep 07 10:40:07.334254 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.334258    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdnc7\" (UniqueName: \"kubernetes.io/projected/82931fe9-8e06-4bf9-aa53-b9de440bcba9-kube-api-access-hdnc7\") pod \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\" (UID: \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\") "
Sep 07 10:40:07.334465 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.334315    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82931fe9-8e06-4bf9-aa53-b9de440bcba9-plugins-config-volume\") pod \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\" (UID: \"82931fe9-8e06-4bf9-aa53-b9de440bcba9\") "
Sep 07 10:40:07.334465 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.334343    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-plugins-config-volume\") pod \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\" (UID: \"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c\") "
Sep 07 10:40:07.334744 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.334719    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82931fe9-8e06-4bf9-aa53-b9de440bcba9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82931fe9-8e06-4bf9-aa53-b9de440bcba9" (UID: "82931fe9-8e06-4bf9-aa53-b9de440bcba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:40:07.334744 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.334731    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" (UID: "0685cd14-5ed0-4791-b6d8-e0272a8d2e6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:40:07.336384 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.336362    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82931fe9-8e06-4bf9-aa53-b9de440bcba9-kube-api-access-hdnc7" (OuterVolumeSpecName: "kube-api-access-hdnc7") pod "82931fe9-8e06-4bf9-aa53-b9de440bcba9" (UID: "82931fe9-8e06-4bf9-aa53-b9de440bcba9"). InnerVolumeSpecName "kube-api-access-hdnc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:40:07.336479 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.336392    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-kube-api-access-hw78c" (OuterVolumeSpecName: "kube-api-access-hw78c") pod "0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" (UID: "0685cd14-5ed0-4791-b6d8-e0272a8d2e6c"). InnerVolumeSpecName "kube-api-access-hw78c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:40:07.435007 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.434983    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw78c\" (UniqueName: \"kubernetes.io/projected/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-kube-api-access-hw78c\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.435007 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.435006    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdnc7\" (UniqueName: \"kubernetes.io/projected/82931fe9-8e06-4bf9-aa53-b9de440bcba9-kube-api-access-hdnc7\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.435149 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.435016    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82931fe9-8e06-4bf9-aa53-b9de440bcba9-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.435149 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.435025    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:07.487945 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.487887    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:07.844615 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.844583    2560 generic.go:356] "Generic (PLEG): container finished" podID="0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" containerID="74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18" exitCode=0
Sep 07 10:40:07.844737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.844627    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"
Sep 07 10:40:07.844737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.844673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47" event={"ID":"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c","Type":"ContainerDied","Data":"74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18"}
Sep 07 10:40:07.844737 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.844726    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47" event={"ID":"0685cd14-5ed0-4791-b6d8-e0272a8d2e6c","Type":"ContainerDied","Data":"244113e99970cd4798e687c2c3eadf1a023d9cef75e1bf4f058f0fadf7bc1449"}
Sep 07 10:40:07.844864 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.844749    2560 scope.go:117] "RemoveContainer" containerID="74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18"
Sep 07 10:40:07.846016 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.845997    2560 generic.go:356] "Generic (PLEG): container finished" podID="82931fe9-8e06-4bf9-aa53-b9de440bcba9" containerID="56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d" exitCode=0
Sep 07 10:40:07.846119 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.846042    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"
Sep 07 10:40:07.846119 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.846092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6" event={"ID":"82931fe9-8e06-4bf9-aa53-b9de440bcba9","Type":"ContainerDied","Data":"56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d"}
Sep 07 10:40:07.846229 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.846122    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6" event={"ID":"82931fe9-8e06-4bf9-aa53-b9de440bcba9","Type":"ContainerDied","Data":"3ae3772ed97bdf987dd8109a66d3fa9f54f3053b8870f09bf8816763f372c76f"}
Sep 07 10:40:07.853805 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.853784    2560 scope.go:117] "RemoveContainer" containerID="74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18"
Sep 07 10:40:07.854116 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:40:07.854098    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18\": container with ID starting with 74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18 not found: ID does not exist" containerID="74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18"
Sep 07 10:40:07.854174 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.854125    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18"} err="failed to get container status \"74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18\": rpc error: code = NotFound desc = could not find container \"74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18\": container with ID starting with 74c4fd1fdcf192b466ab6277b22bef4e98f1e3275f694dc078249570565bdf18 not found: ID does not exist"
Sep 07 10:40:07.854174 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.854142    2560 scope.go:117] "RemoveContainer" containerID="56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d"
Sep 07 10:40:07.861798 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.861781    2560 scope.go:117] "RemoveContainer" containerID="56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d"
Sep 07 10:40:07.862035 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:40:07.862017    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d\": container with ID starting with 56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d not found: ID does not exist" containerID="56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d"
Sep 07 10:40:07.862114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.862043    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d"} err="failed to get container status \"56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d\": rpc error: code = NotFound desc = could not find container \"56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d\": container with ID starting with 56a80e35dc7cbab002e5d39680355ca5cedef3f7fe0702abf02868e198fa3d8d not found: ID does not exist"
Sep 07 10:40:07.867711 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.867690    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"]
Sep 07 10:40:07.871639 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.871619    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-92aa2e-7668dd9b85-gblb6"]
Sep 07 10:40:07.881795 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.881771    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"]
Sep 07 10:40:07.885434 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.885415    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92aa2e-5896448db-snm47"]
Sep 07 10:40:07.894558 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.894537    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" path="/var/lib/kubelet/pods/0685cd14-5ed0-4791-b6d8-e0272a8d2e6c/volumes"
Sep 07 10:40:07.894878 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:07.894866    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82931fe9-8e06-4bf9-aa53-b9de440bcba9" path="/var/lib/kubelet/pods/82931fe9-8e06-4bf9-aa53-b9de440bcba9/volumes"
Sep 07 10:40:12.469655 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:12.469619    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"]
Sep 07 10:40:12.517234 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:12.517203    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:17.489155 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:17.489117    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:40:27.489403 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.489342    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n" podUID="3f65a758-ea00-41be-a602-9b7ab044d7c5" containerName="main" containerID="cri-o://dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340" gracePeriod=45
Sep 07 10:40:27.728878 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.728852    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:40:27.805555 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.805524    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-home\") pod \"3f65a758-ea00-41be-a602-9b7ab044d7c5\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") "
Sep 07 10:40:27.805730 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.805571    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-model-cache\") pod \"3f65a758-ea00-41be-a602-9b7ab044d7c5\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") "
Sep 07 10:40:27.805730 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.805599    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-dshm\") pod \"3f65a758-ea00-41be-a602-9b7ab044d7c5\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") "
Sep 07 10:40:27.805730 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.805666    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjdlr\" (UniqueName: \"kubernetes.io/projected/3f65a758-ea00-41be-a602-9b7ab044d7c5-kube-api-access-bjdlr\") pod \"3f65a758-ea00-41be-a602-9b7ab044d7c5\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") "
Sep 07 10:40:27.805730 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.805706    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f65a758-ea00-41be-a602-9b7ab044d7c5-tls-certs\") pod \"3f65a758-ea00-41be-a602-9b7ab044d7c5\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") "
Sep 07 10:40:27.805938 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.805743    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-tmp-dir\") pod \"3f65a758-ea00-41be-a602-9b7ab044d7c5\" (UID: \"3f65a758-ea00-41be-a602-9b7ab044d7c5\") "
Sep 07 10:40:27.805938 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.805902    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f65a758-ea00-41be-a602-9b7ab044d7c5" (UID: "3f65a758-ea00-41be-a602-9b7ab044d7c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:40:27.805938 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.805917    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-home" (OuterVolumeSpecName: "home") pod "3f65a758-ea00-41be-a602-9b7ab044d7c5" (UID: "3f65a758-ea00-41be-a602-9b7ab044d7c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:40:27.806127 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.806107    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-home\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:27.806127 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.806122    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-model-cache\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:27.806223 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.806150    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f65a758-ea00-41be-a602-9b7ab044d7c5" (UID: "3f65a758-ea00-41be-a602-9b7ab044d7c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:40:27.807923 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.807901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-dshm" (OuterVolumeSpecName: "dshm") pod "3f65a758-ea00-41be-a602-9b7ab044d7c5" (UID: "3f65a758-ea00-41be-a602-9b7ab044d7c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:40:27.808259 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.808232    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f65a758-ea00-41be-a602-9b7ab044d7c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f65a758-ea00-41be-a602-9b7ab044d7c5" (UID: "3f65a758-ea00-41be-a602-9b7ab044d7c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:40:27.808259 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.808247    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f65a758-ea00-41be-a602-9b7ab044d7c5-kube-api-access-bjdlr" (OuterVolumeSpecName: "kube-api-access-bjdlr") pod "3f65a758-ea00-41be-a602-9b7ab044d7c5" (UID: "3f65a758-ea00-41be-a602-9b7ab044d7c5"). InnerVolumeSpecName "kube-api-access-bjdlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:40:27.906940 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.906914    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-dshm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:27.906940 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.906941    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjdlr\" (UniqueName: \"kubernetes.io/projected/3f65a758-ea00-41be-a602-9b7ab044d7c5-kube-api-access-bjdlr\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:27.907102 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.906951    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f65a758-ea00-41be-a602-9b7ab044d7c5-tls-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:27.907102 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.906960    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f65a758-ea00-41be-a602-9b7ab044d7c5-tmp-dir\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:40:27.947829 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.947795    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f65a758-ea00-41be-a602-9b7ab044d7c5" containerID="dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340" exitCode=0
Sep 07 10:40:27.947959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.947845    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"
Sep 07 10:40:27.947959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.947871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n" event={"ID":"3f65a758-ea00-41be-a602-9b7ab044d7c5","Type":"ContainerDied","Data":"dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340"}
Sep 07 10:40:27.947959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.947908    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n" event={"ID":"3f65a758-ea00-41be-a602-9b7ab044d7c5","Type":"ContainerDied","Data":"0a77f55a09227e896f7a5a0a98c01cf03014925897e18f02c80fc49b550358bf"}
Sep 07 10:40:27.947959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.947931    2560 scope.go:117] "RemoveContainer" containerID="dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340"
Sep 07 10:40:27.961530 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.961509    2560 scope.go:117] "RemoveContainer" containerID="dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340"
Sep 07 10:40:27.961823 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:40:27.961804    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340\": container with ID starting with dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340 not found: ID does not exist" containerID="dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340"
Sep 07 10:40:27.961873 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.961833    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340"} err="failed to get container status \"dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340\": rpc error: code = NotFound desc = could not find container \"dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340\": container with ID starting with dab9bc9a3664e5944e5b30a8da0173295cffb56096e7b0effa0af203e0a83340 not found: ID does not exist"
Sep 07 10:40:27.968871 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.968841    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"]
Sep 07 10:40:27.973195 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:27.973166    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-92aa2e/ipp-route-92aa2e-kserve-688db8964c-cnq4n"]
Sep 07 10:40:29.896676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:29.896644    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f65a758-ea00-41be-a602-9b7ab044d7c5" path="/var/lib/kubelet/pods/3f65a758-ea00-41be-a602-9b7ab044d7c5/volumes"
Sep 07 10:40:49.012022 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:49.011990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:49.624507 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:49.624473    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:55.505896 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:55.505854    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:55.581960 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:55.581922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:55.784946 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:55.784911    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:40:56.349461 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:40:56.349428    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:00.598600 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:00.598567    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:06.638624 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:06.638580    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:06.681754 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:06.681723    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"]
Sep 07 10:41:06.682006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:06.681981    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm" podUID="3d3d95c0-fad7-4a54-9604-c0cde9f2ba19" containerName="payload-pre-processing" containerID="cri-o://6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97" gracePeriod=30
Sep 07 10:41:06.742029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:06.741998    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:06.854425 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:06.854387    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:06.925679 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:06.925654    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:41:06.959029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:06.959001    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:07.046857 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.046821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz864\" (UniqueName: \"kubernetes.io/projected/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-kube-api-access-qz864\") pod \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\" (UID: \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\") "
Sep 07 10:41:07.046857 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.046866    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-plugins-config-volume\") pod \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\" (UID: \"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19\") "
Sep 07 10:41:07.047292 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.047269    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d3d95c0-fad7-4a54-9604-c0cde9f2ba19" (UID: "3d3d95c0-fad7-4a54-9604-c0cde9f2ba19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:07.048746 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.048729    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-kube-api-access-qz864" (OuterVolumeSpecName: "kube-api-access-qz864") pod "3d3d95c0-fad7-4a54-9604-c0cde9f2ba19" (UID: "3d3d95c0-fad7-4a54-9604-c0cde9f2ba19"). InnerVolumeSpecName "kube-api-access-qz864". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:07.129434 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.129406    2560 generic.go:356] "Generic (PLEG): container finished" podID="3d3d95c0-fad7-4a54-9604-c0cde9f2ba19" containerID="6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97" exitCode=0
Sep 07 10:41:07.129539 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.129446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm" event={"ID":"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19","Type":"ContainerDied","Data":"6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97"}
Sep 07 10:41:07.129539 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.129455    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"
Sep 07 10:41:07.129539 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.129498    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm" event={"ID":"3d3d95c0-fad7-4a54-9604-c0cde9f2ba19","Type":"ContainerDied","Data":"da86f481069d8b570291d1053100b4b0e321de1a38a7f4c506744a61e1805d6d"}
Sep 07 10:41:07.129539 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.129527    2560 scope.go:117] "RemoveContainer" containerID="6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97"
Sep 07 10:41:07.138110 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.138094    2560 scope.go:117] "RemoveContainer" containerID="6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97"
Sep 07 10:41:07.138384 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:41:07.138367    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97\": container with ID starting with 6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97 not found: ID does not exist" containerID="6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97"
Sep 07 10:41:07.138435 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.138394    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97"} err="failed to get container status \"6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97\": rpc error: code = NotFound desc = could not find container \"6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97\": container with ID starting with 6ff827ea8f27c85cdcdc53567c248480fcb05b11bd20db67c65ea5c9cf39cb97 not found: ID does not exist"
Sep 07 10:41:07.148271 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.148245    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz864\" (UniqueName: \"kubernetes.io/projected/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-kube-api-access-qz864\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:07.148271 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.148275    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:07.160033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.159981    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"]
Sep 07 10:41:07.166937 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.166921    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-125f28-7bc6c88c66-gw5jm"]
Sep 07 10:41:07.350949 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.350922    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:07.895236 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:07.895204    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d3d95c0-fad7-4a54-9604-c0cde9f2ba19" path="/var/lib/kubelet/pods/3d3d95c0-fad7-4a54-9604-c0cde9f2ba19/volumes"
Sep 07 10:41:09.797313 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:09.797281    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:41:13.101276 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:13.101241    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:13.202682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:13.202655    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:13.358921 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:13.358838    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:13.942862 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:13.942826    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:24.312684 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.312642    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"]
Sep 07 10:41:24.313169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.312995    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6" podUID="768ed424-fdfc-4da3-ae52-e7d7a0e1812d" containerName="maas-api" containerID="cri-o://b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b" gracePeriod=30
Sep 07 10:41:24.356168 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.356128    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:24.387107 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.387049    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"]
Sep 07 10:41:24.387278 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.387252    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c" podUID="d1470fc6-7b1c-4579-8195-8e64b1a72157" containerName="payload-pre-processing" containerID="cri-o://99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa" gracePeriod=30
Sep 07 10:41:24.453566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.453525    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:24.553852 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.553817    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:24.622113 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.621990    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:24.642906 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.642878    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:41:24.645621 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.645603    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:41:24.805324 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805288    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-metrics-certs\") pod \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") "
Sep 07 10:41:24.805324 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805332    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1470fc6-7b1c-4579-8195-8e64b1a72157-plugins-config-volume\") pod \"d1470fc6-7b1c-4579-8195-8e64b1a72157\" (UID: \"d1470fc6-7b1c-4579-8195-8e64b1a72157\") "
Sep 07 10:41:24.805583 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805352    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f96nz\" (UniqueName: \"kubernetes.io/projected/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-kube-api-access-f96nz\") pod \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") "
Sep 07 10:41:24.805583 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805376    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls\") pod \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") "
Sep 07 10:41:24.805583 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805421    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-openshift-service-ca\") pod \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\" (UID: \"768ed424-fdfc-4da3-ae52-e7d7a0e1812d\") "
Sep 07 10:41:24.805745 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805612    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzsk6\" (UniqueName: \"kubernetes.io/projected/d1470fc6-7b1c-4579-8195-8e64b1a72157-kube-api-access-zzsk6\") pod \"d1470fc6-7b1c-4579-8195-8e64b1a72157\" (UID: \"d1470fc6-7b1c-4579-8195-8e64b1a72157\") "
Sep 07 10:41:24.805812 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805766    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "768ed424-fdfc-4da3-ae52-e7d7a0e1812d" (UID: "768ed424-fdfc-4da3-ae52-e7d7a0e1812d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:24.805812 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805771    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1470fc6-7b1c-4579-8195-8e64b1a72157-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1470fc6-7b1c-4579-8195-8e64b1a72157" (UID: "d1470fc6-7b1c-4579-8195-8e64b1a72157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:24.805925 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805900    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1470fc6-7b1c-4579-8195-8e64b1a72157-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:24.805983 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.805928    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-openshift-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:24.807558 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.807521    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1470fc6-7b1c-4579-8195-8e64b1a72157-kube-api-access-zzsk6" (OuterVolumeSpecName: "kube-api-access-zzsk6") pod "d1470fc6-7b1c-4579-8195-8e64b1a72157" (UID: "d1470fc6-7b1c-4579-8195-8e64b1a72157"). InnerVolumeSpecName "kube-api-access-zzsk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:24.807670 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.807568    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "768ed424-fdfc-4da3-ae52-e7d7a0e1812d" (UID: "768ed424-fdfc-4da3-ae52-e7d7a0e1812d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:24.807670 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.807569    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-kube-api-access-f96nz" (OuterVolumeSpecName: "kube-api-access-f96nz") pod "768ed424-fdfc-4da3-ae52-e7d7a0e1812d" (UID: "768ed424-fdfc-4da3-ae52-e7d7a0e1812d"). InnerVolumeSpecName "kube-api-access-f96nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:24.807670 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.807587    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "768ed424-fdfc-4da3-ae52-e7d7a0e1812d" (UID: "768ed424-fdfc-4da3-ae52-e7d7a0e1812d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:41:24.907347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.907258    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzsk6\" (UniqueName: \"kubernetes.io/projected/d1470fc6-7b1c-4579-8195-8e64b1a72157-kube-api-access-zzsk6\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:24.907347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.907288    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-metrics-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:24.907347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.907298    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96nz\" (UniqueName: \"kubernetes.io/projected/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-kube-api-access-f96nz\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:24.907347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:24.907307    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/768ed424-fdfc-4da3-ae52-e7d7a0e1812d-maas-api-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:25.075836 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.075806    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:25.216876 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.216787    2560 generic.go:356] "Generic (PLEG): container finished" podID="d1470fc6-7b1c-4579-8195-8e64b1a72157" containerID="99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa" exitCode=0
Sep 07 10:41:25.216876 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.216829    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"
Sep 07 10:41:25.217113 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.216872    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c" event={"ID":"d1470fc6-7b1c-4579-8195-8e64b1a72157","Type":"ContainerDied","Data":"99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa"}
Sep 07 10:41:25.217113 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.216925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c" event={"ID":"d1470fc6-7b1c-4579-8195-8e64b1a72157","Type":"ContainerDied","Data":"f61c40f6bf2cc2e59715e379c44288265b77adb8313c145929d0009e5289a4c5"}
Sep 07 10:41:25.217113 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.216948    2560 scope.go:117] "RemoveContainer" containerID="99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa"
Sep 07 10:41:25.218291 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.218270    2560 generic.go:356] "Generic (PLEG): container finished" podID="768ed424-fdfc-4da3-ae52-e7d7a0e1812d" containerID="b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b" exitCode=0
Sep 07 10:41:25.218412 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.218303    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6" event={"ID":"768ed424-fdfc-4da3-ae52-e7d7a0e1812d","Type":"ContainerDied","Data":"b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b"}
Sep 07 10:41:25.218412 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.218318    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"
Sep 07 10:41:25.218412 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.218335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6" event={"ID":"768ed424-fdfc-4da3-ae52-e7d7a0e1812d","Type":"ContainerDied","Data":"e1ff715138fd735428995ea1f2df6a0be0fb36b214f9948d4ac95a9b092e192f"}
Sep 07 10:41:25.227809 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.227784    2560 scope.go:117] "RemoveContainer" containerID="99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa"
Sep 07 10:41:25.228527 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:41:25.228502    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa\": container with ID starting with 99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa not found: ID does not exist" containerID="99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa"
Sep 07 10:41:25.228616 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.228538    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa"} err="failed to get container status \"99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa\": rpc error: code = NotFound desc = could not find container \"99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa\": container with ID starting with 99c6721f6e0654ecf12fce856ee725e1ab2c6952d7f4376a848ab5076ec942aa not found: ID does not exist"
Sep 07 10:41:25.228616 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.228559    2560 scope.go:117] "RemoveContainer" containerID="b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b"
Sep 07 10:41:25.236674 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.236658    2560 scope.go:117] "RemoveContainer" containerID="b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b"
Sep 07 10:41:25.236914 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:41:25.236900    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b\": container with ID starting with b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b not found: ID does not exist" containerID="b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b"
Sep 07 10:41:25.236967 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.236920    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b"} err="failed to get container status \"b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b\": rpc error: code = NotFound desc = could not find container \"b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b\": container with ID starting with b55da9179cce2c5fbe2e4fcb05c564ecdf7d721646237468190c0c4b97e1de7b not found: ID does not exist"
Sep 07 10:41:25.264484 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.264457    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"]
Sep 07 10:41:25.268671 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.268652    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a577c-775b9b67c6-57lw6"]
Sep 07 10:41:25.293724 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.293672    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"]
Sep 07 10:41:25.297419 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.297400    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0a577c-855df84646-2ff8c"]
Sep 07 10:41:25.895270 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.895240    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="768ed424-fdfc-4da3-ae52-e7d7a0e1812d" path="/var/lib/kubelet/pods/768ed424-fdfc-4da3-ae52-e7d7a0e1812d/volumes"
Sep 07 10:41:25.895761 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:25.895741    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1470fc6-7b1c-4579-8195-8e64b1a72157" path="/var/lib/kubelet/pods/d1470fc6-7b1c-4579-8195-8e64b1a72157/volumes"
Sep 07 10:41:27.699543 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:27.699458    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:41:31.147384 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.147346    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:31.246465 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.246429    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:31.454218 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.454131    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:31.824347 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824307    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"]
Sep 07 10:41:31.824639 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824627    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f65a758-ea00-41be-a602-9b7ab044d7c5" containerName="main"
Sep 07 10:41:31.824682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824640    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f65a758-ea00-41be-a602-9b7ab044d7c5" containerName="main"
Sep 07 10:41:31.824682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824650    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="768ed424-fdfc-4da3-ae52-e7d7a0e1812d" containerName="maas-api"
Sep 07 10:41:31.824682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824655    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="768ed424-fdfc-4da3-ae52-e7d7a0e1812d" containerName="maas-api"
Sep 07 10:41:31.824682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824662    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d3d95c0-fad7-4a54-9604-c0cde9f2ba19" containerName="payload-pre-processing"
Sep 07 10:41:31.824682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824668    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d3d95c0-fad7-4a54-9604-c0cde9f2ba19" containerName="payload-pre-processing"
Sep 07 10:41:31.824682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824675    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac78cbd1-4239-451f-8844-c80ff87b8f20" containerName="payload-processing"
Sep 07 10:41:31.824682 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824679    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac78cbd1-4239-451f-8844-c80ff87b8f20" containerName="payload-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824689    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1470fc6-7b1c-4579-8195-8e64b1a72157" containerName="payload-pre-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824694    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1470fc6-7b1c-4579-8195-8e64b1a72157" containerName="payload-pre-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824700    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82931fe9-8e06-4bf9-aa53-b9de440bcba9" containerName="payload-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824705    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82931fe9-8e06-4bf9-aa53-b9de440bcba9" containerName="payload-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824715    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" containerName="payload-pre-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824721    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" containerName="payload-pre-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824770    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="768ed424-fdfc-4da3-ae52-e7d7a0e1812d" containerName="maas-api"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824777    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d3d95c0-fad7-4a54-9604-c0cde9f2ba19" containerName="payload-pre-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824782    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f65a758-ea00-41be-a602-9b7ab044d7c5" containerName="main"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824787    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac78cbd1-4239-451f-8844-c80ff87b8f20" containerName="payload-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824793    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0685cd14-5ed0-4791-b6d8-e0272a8d2e6c" containerName="payload-pre-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824799    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1470fc6-7b1c-4579-8195-8e64b1a72157" containerName="payload-pre-processing"
Sep 07 10:41:31.824890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.824807    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82931fe9-8e06-4bf9-aa53-b9de440bcba9" containerName="payload-processing"
Sep 07 10:41:31.829103 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.829086    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"
Sep 07 10:41:31.833104 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.833085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 10:41:31.833232 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.833213    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\""
Sep 07 10:41:31.872467 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.872446    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"]
Sep 07 10:41:31.965254 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:31.965228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2815d212-5d18-4d8c-baa5-4d93c00b7d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj\" (UID: \"2815d212-5d18-4d8c-baa5-4d93c00b7d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"
Sep 07 10:41:32.049674 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:32.049642    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:32.065751 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:32.065722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2815d212-5d18-4d8c-baa5-4d93c00b7d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj\" (UID: \"2815d212-5d18-4d8c-baa5-4d93c00b7d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"
Sep 07 10:41:32.066302 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:32.066287    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2815d212-5d18-4d8c-baa5-4d93c00b7d99-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj\" (UID: \"2815d212-5d18-4d8c-baa5-4d93c00b7d99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"
Sep 07 10:41:32.138659 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:32.138608    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"
Sep 07 10:41:32.410903 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:32.410831    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"]
Sep 07 10:41:32.414372 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:41:32.414348    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2815d212_5d18_4d8c_baa5_4d93c00b7d99.slice/crio-eb191728acbe80d5f7b66f34e2609eedc0a0c6aad7b1b3a4d2d6759b4fd8fd5a WatchSource:0}: Error finding container eb191728acbe80d5f7b66f34e2609eedc0a0c6aad7b1b3a4d2d6759b4fd8fd5a: Status 404 returned error can't find the container with id eb191728acbe80d5f7b66f34e2609eedc0a0c6aad7b1b3a4d2d6759b4fd8fd5a
Sep 07 10:41:33.256439 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:33.256401    2560 generic.go:356] "Generic (PLEG): container finished" podID="2815d212-5d18-4d8c-baa5-4d93c00b7d99" containerID="7310eb2e77707a2eeef72d63b526902dacd10bb8dfab53954bfd08799446241a" exitCode=0
Sep 07 10:41:33.256633 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:33.256486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj" event={"ID":"2815d212-5d18-4d8c-baa5-4d93c00b7d99","Type":"ContainerDied","Data":"7310eb2e77707a2eeef72d63b526902dacd10bb8dfab53954bfd08799446241a"}
Sep 07 10:41:33.256633 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:33.256536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj" event={"ID":"2815d212-5d18-4d8c-baa5-4d93c00b7d99","Type":"ContainerStarted","Data":"eb191728acbe80d5f7b66f34e2609eedc0a0c6aad7b1b3a4d2d6759b4fd8fd5a"}
Sep 07 10:41:34.387553 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:34.387525    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"
Sep 07 10:41:34.491663 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:34.491630    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2815d212-5d18-4d8c-baa5-4d93c00b7d99-maas-api-service-ca\") pod \"2815d212-5d18-4d8c-baa5-4d93c00b7d99\" (UID: \"2815d212-5d18-4d8c-baa5-4d93c00b7d99\") "
Sep 07 10:41:34.492046 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:34.492024    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2815d212-5d18-4d8c-baa5-4d93c00b7d99-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2815d212-5d18-4d8c-baa5-4d93c00b7d99" (UID: "2815d212-5d18-4d8c-baa5-4d93c00b7d99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:34.593133 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:34.593098    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2815d212-5d18-4d8c-baa5-4d93c00b7d99-maas-api-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:35.266908 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:35.266878    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"
Sep 07 10:41:35.267055 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:35.266875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj" event={"ID":"2815d212-5d18-4d8c-baa5-4d93c00b7d99","Type":"ContainerDied","Data":"eb191728acbe80d5f7b66f34e2609eedc0a0c6aad7b1b3a4d2d6759b4fd8fd5a"}
Sep 07 10:41:35.267055 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:35.266998    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb191728acbe80d5f7b66f34e2609eedc0a0c6aad7b1b3a4d2d6759b4fd8fd5a"
Sep 07 10:41:36.832554 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:36.832514    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:41.994344 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:41.994308    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:42.649833 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.649798    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:42.703806 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.703769    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"]
Sep 07 10:41:42.704065 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.704009    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q" podUID="7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d" containerName="payload-processing" containerID="cri-o://88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81" gracePeriod=30
Sep 07 10:41:42.751267 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.751238    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:42.849720 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.849690    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:42.901748 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.901719    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"]
Sep 07 10:41:42.904767 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.904744    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-93e41e-74bqj"]
Sep 07 10:41:42.915829 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.915801    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:42.945169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.945151    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:41:42.954893 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.954873    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmbp9\" (UniqueName: \"kubernetes.io/projected/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-kube-api-access-kmbp9\") pod \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\" (UID: \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\") "
Sep 07 10:41:42.954997 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.954906    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-plugins-config-volume\") pod \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\" (UID: \"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d\") "
Sep 07 10:41:42.955310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.955290    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d" (UID: "7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:41:42.956828 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:42.956807    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-kube-api-access-kmbp9" (OuterVolumeSpecName: "kube-api-access-kmbp9") pod "7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d" (UID: "7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d"). InnerVolumeSpecName "kube-api-access-kmbp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:41:43.055430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.055401    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-plugins-config-volume\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:43.055430 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.055427    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmbp9\" (UniqueName: \"kubernetes.io/projected/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d-kube-api-access-kmbp9\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:41:43.277084 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.277039    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:43.305081 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.305037    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d" containerID="88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81" exitCode=0
Sep 07 10:41:43.305239 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.305108    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"
Sep 07 10:41:43.305282 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.305099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q" event={"ID":"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d","Type":"ContainerDied","Data":"88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81"}
Sep 07 10:41:43.305317 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.305305    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q" event={"ID":"7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d","Type":"ContainerDied","Data":"7114074a2ea01e82a270998a62d412ca10f65e08768fb6a802a2c70a3c7e1b33"}
Sep 07 10:41:43.305355 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.305336    2560 scope.go:117] "RemoveContainer" containerID="88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81"
Sep 07 10:41:43.314483 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.314470    2560 scope.go:117] "RemoveContainer" containerID="88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81"
Sep 07 10:41:43.314753 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:41:43.314734    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81\": container with ID starting with 88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81 not found: ID does not exist" containerID="88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81"
Sep 07 10:41:43.314826 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.314759    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81"} err="failed to get container status \"88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81\": rpc error: code = NotFound desc = could not find container \"88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81\": container with ID starting with 88631c38c270c9590735da93c003c074a94ed622d21296258520c94f52b18a81 not found: ID does not exist"
Sep 07 10:41:43.327910 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.327883    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"]
Sep 07 10:41:43.334571 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.334545    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-93e41e-7648cb4bb7-hm99q"]
Sep 07 10:41:43.894743 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.894714    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2815d212-5d18-4d8c-baa5-4d93c00b7d99" path="/var/lib/kubelet/pods/2815d212-5d18-4d8c-baa5-4d93c00b7d99/volumes"
Sep 07 10:41:43.895081 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:43.895057    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d" path="/var/lib/kubelet/pods/7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d/volumes"
Sep 07 10:41:45.295667 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:45.295633    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:41:49.906555 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:49.906439    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:50.009958 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:50.009925    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:50.202959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:50.202873    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:41:54.592695 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:41:54.592659    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:42:34.099772 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:42:34.099742    2560 scope.go:117] "RemoveContainer" containerID="afb26f0e090bcff6dca972dd8def585fac702484ea8583754c4ef2b796ecd76f"
Sep 07 10:43:14.377956 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:14.377921    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:43:15.398125 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:15.398092    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:43:15.710612 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:15.710504    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:43:17.184919 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:17.184885    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:43:37.842412 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:37.842379    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:43:37.932659 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:37.932620    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:43:41.308061 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:41.308024    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:43:49.694091 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:49.694025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:43:56.516775 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:43:56.516739    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:44:11.522160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:44:11.522123    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:44:19.425753 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:44:19.425709    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:45:00.138559 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.138462    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r"]
Sep 07 10:45:00.139158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.138951    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2815d212-5d18-4d8c-baa5-4d93c00b7d99" containerName="revoke-keys"
Sep 07 10:45:00.139158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.138970    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2815d212-5d18-4d8c-baa5-4d93c00b7d99" containerName="revoke-keys"
Sep 07 10:45:00.139158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.138997    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d" containerName="payload-processing"
Sep 07 10:45:00.139158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.139006    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d" containerName="payload-processing"
Sep 07 10:45:00.139158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.139110    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2815d212-5d18-4d8c-baa5-4d93c00b7d99" containerName="revoke-keys"
Sep 07 10:45:00.139158 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.139126    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7f9cec-a5ad-4df1-97c4-3d5bcd8cef6d" containerName="payload-processing"
Sep 07 10:45:00.142295 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.142274    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r"
Sep 07 10:45:00.144888 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.144864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6766n\""
Sep 07 10:45:00.148833 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.148807    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r"]
Sep 07 10:45:00.152618 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.152595    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r"
Sep 07 10:45:00.433452 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:00.433426    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r"]
Sep 07 10:45:00.436041 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:45:00.435979    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f62a6b_4c91_4a25_a570_5f54bc98fae3.slice/crio-48c0055de061b3ae09b1c82661d8d5fa375922ea7f395388de84334d1af3889c WatchSource:0}: Error finding container 48c0055de061b3ae09b1c82661d8d5fa375922ea7f395388de84334d1af3889c: Status 404 returned error can't find the container with id 48c0055de061b3ae09b1c82661d8d5fa375922ea7f395388de84334d1af3889c
Sep 07 10:45:01.195455 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:01.195416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r" event={"ID":"d8f62a6b-4c91-4a25-a570-5f54bc98fae3","Type":"ContainerStarted","Data":"5f00c2ee1613b29a81194d1d96bc9ece25e72f059464fd1b4a36dcc52d31b4a4"}
Sep 07 10:45:01.195933 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:01.195472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r" event={"ID":"d8f62a6b-4c91-4a25-a570-5f54bc98fae3","Type":"ContainerStarted","Data":"48c0055de061b3ae09b1c82661d8d5fa375922ea7f395388de84334d1af3889c"}
Sep 07 10:45:01.217924 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:01.217870    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r" podStartSLOduration=1.217854321 podStartE2EDuration="1.217854321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:45:01.215609588 +0000 UTC m=+2007.764124573" watchObservedRunningTime="2026-09-07 10:45:01.217854321 +0000 UTC m=+2007.766369308"
Sep 07 10:45:02.201242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:02.201206    2560 generic.go:356] "Generic (PLEG): container finished" podID="d8f62a6b-4c91-4a25-a570-5f54bc98fae3" containerID="5f00c2ee1613b29a81194d1d96bc9ece25e72f059464fd1b4a36dcc52d31b4a4" exitCode=0
Sep 07 10:45:02.201636 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:02.201281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r" event={"ID":"d8f62a6b-4c91-4a25-a570-5f54bc98fae3","Type":"ContainerDied","Data":"5f00c2ee1613b29a81194d1d96bc9ece25e72f059464fd1b4a36dcc52d31b4a4"}
Sep 07 10:45:03.336532 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:03.336506    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r"
Sep 07 10:45:04.212190 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:04.212108    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r"
Sep 07 10:45:04.212326 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:04.212109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29812965-6t97r" event={"ID":"d8f62a6b-4c91-4a25-a570-5f54bc98fae3","Type":"ContainerDied","Data":"48c0055de061b3ae09b1c82661d8d5fa375922ea7f395388de84334d1af3889c"}
Sep 07 10:45:04.212326 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:04.212234    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48c0055de061b3ae09b1c82661d8d5fa375922ea7f395388de84334d1af3889c"
Sep 07 10:45:40.437645 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:40.437606    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:45:48.498730 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:48.498697    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:45:54.318850 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:45:54.318812    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:46:01.698610 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:01.698523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:46:03.793123 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.793089    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"]
Sep 07 10:46:03.793526 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.793441    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f62a6b-4c91-4a25-a570-5f54bc98fae3" containerName="cleanup"
Sep 07 10:46:03.793526 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.793454    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f62a6b-4c91-4a25-a570-5f54bc98fae3" containerName="cleanup"
Sep 07 10:46:03.793526 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.793506    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f62a6b-4c91-4a25-a570-5f54bc98fae3" containerName="cleanup"
Sep 07 10:46:03.796558 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.796541    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:03.799220 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.799195    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-49bb7129-kserve-self-signed-certs\""
Sep 07 10:46:03.807179 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.807154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"]
Sep 07 10:46:03.905311 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.905272    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:46:03.946034 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.945992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-model-cache\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:03.946242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.946056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-home\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:03.946242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.946103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tls-certs\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:03.946242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.946135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-dshm\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:03.946242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.946157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tmp-dir\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:03.946242 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:03.946222    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svd7m\" (UniqueName: \"kubernetes.io/projected/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-kube-api-access-svd7m\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.046989 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.046888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-model-cache\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.046989 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.046953    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-home\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.046989 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.046971    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tls-certs\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.047324 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.047000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-dshm\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.047324 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.047014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tmp-dir\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.047324 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.047118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svd7m\" (UniqueName: \"kubernetes.io/projected/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-kube-api-access-svd7m\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.047470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.047356    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-model-cache\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.047470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.047402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tmp-dir\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.047470 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.047421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-home\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.049357 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.049325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-dshm\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.049646 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.049626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tls-certs\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.056015 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.055989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svd7m\" (UniqueName: \"kubernetes.io/projected/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-kube-api-access-svd7m\") pod \"e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") " pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.106033 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.105996    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:04.377994 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.377965    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"]
Sep 07 10:46:04.380811 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:46:04.380782    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc128f9f3_d152_48d8_90c4_8a6d89c1dbff.slice/crio-ccbf37f83dd09f8b4dcde467d49324af8e9c4844b4f183aa753b2a3e53c48ce3 WatchSource:0}: Error finding container ccbf37f83dd09f8b4dcde467d49324af8e9c4844b4f183aa753b2a3e53c48ce3: Status 404 returned error can't find the container with id ccbf37f83dd09f8b4dcde467d49324af8e9c4844b4f183aa753b2a3e53c48ce3
Sep 07 10:46:04.489914 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.489871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs" event={"ID":"c128f9f3-d152-48d8-90c4-8a6d89c1dbff","Type":"ContainerStarted","Data":"cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92"}
Sep 07 10:46:04.489914 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.489913    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs" event={"ID":"c128f9f3-d152-48d8-90c4-8a6d89c1dbff","Type":"ContainerStarted","Data":"ccbf37f83dd09f8b4dcde467d49324af8e9c4844b4f183aa753b2a3e53c48ce3"}
Sep 07 10:46:04.507485 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:04.507435    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs" podStartSLOduration=1.507417588 podStartE2EDuration="1.507417588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:46:04.505950393 +0000 UTC m=+2071.054465378" watchObservedRunningTime="2026-09-07 10:46:04.507417588 +0000 UTC m=+2071.055932576"
Sep 07 10:46:10.698648 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:10.698617    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:46:14.106954 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:14.106915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:14.106954 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:14.106964    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:14.120685 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:14.120658    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:14.556784 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:14.556744    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:46:18.629144 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:18.629102    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:46:21.191779 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:46:21.191741    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:00.251305 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:00.251269    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"]
Sep 07 10:47:00.273594 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:00.273555    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:47:02.348181 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:02.348144    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:47:02.495025 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:02.494984    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:11.490897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:11.490867    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:15.270986 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.270943    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs" podUID="c128f9f3-d152-48d8-90c4-8a6d89c1dbff" containerName="main" containerID="cri-o://cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92" gracePeriod=45
Sep 07 10:47:15.510914 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.510888    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:47:15.602850 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.602824    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tls-certs\") pod \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") "
Sep 07 10:47:15.603038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.602861    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-dshm\") pod \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") "
Sep 07 10:47:15.603038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.602904    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tmp-dir\") pod \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") "
Sep 07 10:47:15.603038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.602951    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-home\") pod \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") "
Sep 07 10:47:15.603038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.602967    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svd7m\" (UniqueName: \"kubernetes.io/projected/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-kube-api-access-svd7m\") pod \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") "
Sep 07 10:47:15.603038 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.602998    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-model-cache\") pod \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\" (UID: \"c128f9f3-d152-48d8-90c4-8a6d89c1dbff\") "
Sep 07 10:47:15.603337 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.603217    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c128f9f3-d152-48d8-90c4-8a6d89c1dbff" (UID: "c128f9f3-d152-48d8-90c4-8a6d89c1dbff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:15.603337 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.603238    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-home" (OuterVolumeSpecName: "home") pod "c128f9f3-d152-48d8-90c4-8a6d89c1dbff" (UID: "c128f9f3-d152-48d8-90c4-8a6d89c1dbff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:15.603337 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.603313    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-model-cache" (OuterVolumeSpecName: "model-cache") pod "c128f9f3-d152-48d8-90c4-8a6d89c1dbff" (UID: "c128f9f3-d152-48d8-90c4-8a6d89c1dbff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:15.605035 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.605016    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-dshm" (OuterVolumeSpecName: "dshm") pod "c128f9f3-d152-48d8-90c4-8a6d89c1dbff" (UID: "c128f9f3-d152-48d8-90c4-8a6d89c1dbff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 10:47:15.605122 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.605019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c128f9f3-d152-48d8-90c4-8a6d89c1dbff" (UID: "c128f9f3-d152-48d8-90c4-8a6d89c1dbff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:47:15.605170 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.605125    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-kube-api-access-svd7m" (OuterVolumeSpecName: "kube-api-access-svd7m") pod "c128f9f3-d152-48d8-90c4-8a6d89c1dbff" (UID: "c128f9f3-d152-48d8-90c4-8a6d89c1dbff"). InnerVolumeSpecName "kube-api-access-svd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:47:15.703907 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.703869    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-home\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:15.703907 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.703902    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svd7m\" (UniqueName: \"kubernetes.io/projected/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-kube-api-access-svd7m\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:15.703907 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.703914    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-model-cache\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:15.704169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.703922    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tls-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:15.704169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.703930    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-dshm\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:15.704169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.703937    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c128f9f3-d152-48d8-90c4-8a6d89c1dbff-tmp-dir\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:15.837939 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.837903    2560 generic.go:356] "Generic (PLEG): container finished" podID="c128f9f3-d152-48d8-90c4-8a6d89c1dbff" containerID="cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92" exitCode=0
Sep 07 10:47:15.838105 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.837950    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"
Sep 07 10:47:15.838105 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.837994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs" event={"ID":"c128f9f3-d152-48d8-90c4-8a6d89c1dbff","Type":"ContainerDied","Data":"cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92"}
Sep 07 10:47:15.838105 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.838038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs" event={"ID":"c128f9f3-d152-48d8-90c4-8a6d89c1dbff","Type":"ContainerDied","Data":"ccbf37f83dd09f8b4dcde467d49324af8e9c4844b4f183aa753b2a3e53c48ce3"}
Sep 07 10:47:15.838105 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.838059    2560 scope.go:117] "RemoveContainer" containerID="cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92"
Sep 07 10:47:15.847274 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.847256    2560 scope.go:117] "RemoveContainer" containerID="cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92"
Sep 07 10:47:15.847512 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:47:15.847496    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92\": container with ID starting with cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92 not found: ID does not exist" containerID="cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92"
Sep 07 10:47:15.847552 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.847521    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92"} err="failed to get container status \"cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92\": rpc error: code = NotFound desc = could not find container \"cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92\": container with ID starting with cf4fbc5739bbfde0bbc64a6c2043ff08b6982ba624c73f691cf199eb2dbbed92 not found: ID does not exist"
Sep 07 10:47:15.859961 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.859902    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"]
Sep 07 10:47:15.861353 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.861336    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-49bb7129-kserve-67d4dc798c-z2hcs"]
Sep 07 10:47:15.894499 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:15.894475    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c128f9f3-d152-48d8-90c4-8a6d89c1dbff" path="/var/lib/kubelet/pods/c128f9f3-d152-48d8-90c4-8a6d89c1dbff/volumes"
Sep 07 10:47:18.387523 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:18.387489    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:25.818410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.818375    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"]
Sep 07 10:47:25.818859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.818762    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c128f9f3-d152-48d8-90c4-8a6d89c1dbff" containerName="main"
Sep 07 10:47:25.818859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.818776    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c128f9f3-d152-48d8-90c4-8a6d89c1dbff" containerName="main"
Sep 07 10:47:25.818859 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.818832    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c128f9f3-d152-48d8-90c4-8a6d89c1dbff" containerName="main"
Sep 07 10:47:25.823235 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.823215    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.827114 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.827094    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 10:47:25.827244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.827094    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 10:47:25.827244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.827094    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 10:47:25.827244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.827132    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 10:47:25.827244 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.827145    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2ldcm\""
Sep 07 10:47:25.833022 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.832947    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"]
Sep 07 10:47:25.886373 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.886337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf94n\" (UniqueName: \"kubernetes.io/projected/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-kube-api-access-gf94n\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.886566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.886418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-metrics-certs\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.886566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.886456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-maas-api-tls\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.886566 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.886518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-openshift-service-ca\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.987459 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.987421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-openshift-service-ca\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.987646 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.987482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf94n\" (UniqueName: \"kubernetes.io/projected/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-kube-api-access-gf94n\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.987646 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.987538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-metrics-certs\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.987646 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.987565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-maas-api-tls\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.988272 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.988249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-openshift-service-ca\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.990164 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.990136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-metrics-certs\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.990258 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.990142    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-maas-api-tls\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:25.995901 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:25.995880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf94n\" (UniqueName: \"kubernetes.io/projected/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-kube-api-access-gf94n\") pod \"maas-api-7596b5dfb4-79wwr\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") " pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:26.136550 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:26.136437    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:26.413415 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:26.413347    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"]
Sep 07 10:47:26.415561 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:47:26.415533    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf9f45f_f69d_4f86_9a54_df8aca3f438d.slice/crio-64154a734ddc6c32f9c5230a326e7d6abb537a5d840f8c13ad0148d4b2917ccd WatchSource:0}: Error finding container 64154a734ddc6c32f9c5230a326e7d6abb537a5d840f8c13ad0148d4b2917ccd: Status 404 returned error can't find the container with id 64154a734ddc6c32f9c5230a326e7d6abb537a5d840f8c13ad0148d4b2917ccd
Sep 07 10:47:26.417006 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:26.416985    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 10:47:26.889152 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:26.889038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr" event={"ID":"fbf9f45f-f69d-4f86-9a54-df8aca3f438d","Type":"ContainerStarted","Data":"28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96"}
Sep 07 10:47:26.889520 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:26.889213    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:26.889520 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:26.889276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr" event={"ID":"fbf9f45f-f69d-4f86-9a54-df8aca3f438d","Type":"ContainerStarted","Data":"64154a734ddc6c32f9c5230a326e7d6abb537a5d840f8c13ad0148d4b2917ccd"}
Sep 07 10:47:26.906029 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:26.905981    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr" podStartSLOduration=1.499634426 podStartE2EDuration="1.905967909s" totalImagesPullingTime="406.333483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:47:26.905013346 +0000 UTC m=+2153.453528338" watchObservedRunningTime="2026-09-07 10:47:26.905967909 +0000 UTC m=+2153.454482896"
Sep 07 10:47:27.091542 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:27.091464    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:32.901305 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:32.901272    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:33.698169 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:33.698139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:33.857125 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:33.857093    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"]
Sep 07 10:47:33.860721 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:33.860706    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:33.870977 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:33.870955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"]
Sep 07 10:47:33.955884 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:33.955819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4fb05f-510d-4d76-9410-a932d68305ee-maas-api-tls\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:33.955884 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:33.955863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4fb05f-510d-4d76-9410-a932d68305ee-openshift-service-ca\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:33.955884 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:33.955882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthjc\" (UniqueName: \"kubernetes.io/projected/5c4fb05f-510d-4d76-9410-a932d68305ee-kube-api-access-mthjc\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:33.956341 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:33.956014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c4fb05f-510d-4d76-9410-a932d68305ee-metrics-certs\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.057331 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.057305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c4fb05f-510d-4d76-9410-a932d68305ee-metrics-certs\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.057438 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.057348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4fb05f-510d-4d76-9410-a932d68305ee-maas-api-tls\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.057438 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.057377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4fb05f-510d-4d76-9410-a932d68305ee-openshift-service-ca\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.057526 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.057481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthjc\" (UniqueName: \"kubernetes.io/projected/5c4fb05f-510d-4d76-9410-a932d68305ee-kube-api-access-mthjc\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.057898 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.057879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c4fb05f-510d-4d76-9410-a932d68305ee-openshift-service-ca\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.059637 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.059619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c4fb05f-510d-4d76-9410-a932d68305ee-maas-api-tls\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.059706 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.059665    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c4fb05f-510d-4d76-9410-a932d68305ee-metrics-certs\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.069905 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.069884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthjc\" (UniqueName: \"kubernetes.io/projected/5c4fb05f-510d-4d76-9410-a932d68305ee-kube-api-access-mthjc\") pod \"maas-api-6b9564d484-4m8dq\" (UID: \"5c4fb05f-510d-4d76-9410-a932d68305ee\") " pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.148912 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.148889    2560 scope.go:117] "RemoveContainer" containerID="7310eb2e77707a2eeef72d63b526902dacd10bb8dfab53954bfd08799446241a"
Sep 07 10:47:34.171630 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.171615    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.440577 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.440555    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"]
Sep 07 10:47:34.442635 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:47:34.442614    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4fb05f_510d_4d76_9410_a932d68305ee.slice/crio-349a312513bcbd2c4984ba6829e1475d137b7abd34f7c34db2b5da18933404e7 WatchSource:0}: Error finding container 349a312513bcbd2c4984ba6829e1475d137b7abd34f7c34db2b5da18933404e7: Status 404 returned error can't find the container with id 349a312513bcbd2c4984ba6829e1475d137b7abd34f7c34db2b5da18933404e7
Sep 07 10:47:34.933168 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.933137    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq" event={"ID":"5c4fb05f-510d-4d76-9410-a932d68305ee","Type":"ContainerStarted","Data":"96a654f62150b0570b1d1551eda28b296141999092e1fd05a0a7c988b289d29d"}
Sep 07 10:47:34.933297 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.933185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq" event={"ID":"5c4fb05f-510d-4d76-9410-a932d68305ee","Type":"ContainerStarted","Data":"349a312513bcbd2c4984ba6829e1475d137b7abd34f7c34db2b5da18933404e7"}
Sep 07 10:47:34.933297 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.933240    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:34.955948 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:34.955909    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq" podStartSLOduration=1.57737024 podStartE2EDuration="1.955896442s" totalImagesPullingTime="378.526202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 10:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:47:34.954126257 +0000 UTC m=+2161.502641245" watchObservedRunningTime="2026-09-07 10:47:34.955896442 +0000 UTC m=+2161.504411428"
Sep 07 10:47:40.942595 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:40.942561    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b9564d484-4m8dq"
Sep 07 10:47:40.997326 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:40.997295    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"]
Sep 07 10:47:40.997504 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:40.997480    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr" podUID="fbf9f45f-f69d-4f86-9a54-df8aca3f438d" containerName="maas-api" containerID="cri-o://28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96" gracePeriod=30
Sep 07 10:47:41.238934 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.238909    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:41.324050 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.324014    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-openshift-service-ca\") pod \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") "
Sep 07 10:47:41.324255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.324122    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-maas-api-tls\") pod \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") "
Sep 07 10:47:41.324255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.324191    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf94n\" (UniqueName: \"kubernetes.io/projected/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-kube-api-access-gf94n\") pod \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") "
Sep 07 10:47:41.324255 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.324233    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-metrics-certs\") pod \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\" (UID: \"fbf9f45f-f69d-4f86-9a54-df8aca3f438d\") "
Sep 07 10:47:41.324468 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.324438    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fbf9f45f-f69d-4f86-9a54-df8aca3f438d" (UID: "fbf9f45f-f69d-4f86-9a54-df8aca3f438d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 10:47:41.324542 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.324525    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-openshift-service-ca\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:41.326344 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.326321    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fbf9f45f-f69d-4f86-9a54-df8aca3f438d" (UID: "fbf9f45f-f69d-4f86-9a54-df8aca3f438d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:47:41.326410 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.326342    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fbf9f45f-f69d-4f86-9a54-df8aca3f438d" (UID: "fbf9f45f-f69d-4f86-9a54-df8aca3f438d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 10:47:41.326456 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.326431    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-kube-api-access-gf94n" (OuterVolumeSpecName: "kube-api-access-gf94n") pod "fbf9f45f-f69d-4f86-9a54-df8aca3f438d" (UID: "fbf9f45f-f69d-4f86-9a54-df8aca3f438d"). InnerVolumeSpecName "kube-api-access-gf94n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 10:47:41.425759 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.425733    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-maas-api-tls\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:41.425759 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.425756    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf94n\" (UniqueName: \"kubernetes.io/projected/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-kube-api-access-gf94n\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:41.425923 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.425766    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbf9f45f-f69d-4f86-9a54-df8aca3f438d-metrics-certs\") on node \"ip-10-0-128-191.ec2.internal\" DevicePath \"\""
Sep 07 10:47:41.967185 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.967154    2560 generic.go:356] "Generic (PLEG): container finished" podID="fbf9f45f-f69d-4f86-9a54-df8aca3f438d" containerID="28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96" exitCode=0
Sep 07 10:47:41.967545 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.967201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr" event={"ID":"fbf9f45f-f69d-4f86-9a54-df8aca3f438d","Type":"ContainerDied","Data":"28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96"}
Sep 07 10:47:41.967545 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.967236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr" event={"ID":"fbf9f45f-f69d-4f86-9a54-df8aca3f438d","Type":"ContainerDied","Data":"64154a734ddc6c32f9c5230a326e7d6abb537a5d840f8c13ad0148d4b2917ccd"}
Sep 07 10:47:41.967545 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.967255    2560 scope.go:117] "RemoveContainer" containerID="28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96"
Sep 07 10:47:41.967545 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.967204    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"
Sep 07 10:47:41.975897 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.975877    2560 scope.go:117] "RemoveContainer" containerID="28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96"
Sep 07 10:47:41.976162 ip-10-0-128-191 kubenswrapper[2560]: E0907 10:47:41.976141    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96\": container with ID starting with 28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96 not found: ID does not exist" containerID="28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96"
Sep 07 10:47:41.976215 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.976174    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96"} err="failed to get container status \"28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96\": rpc error: code = NotFound desc = could not find container \"28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96\": container with ID starting with 28f6ee44c7d055b7afdfc1ff6330deec9c2948e2ceb59cc7f5d5814fb4fc0e96 not found: ID does not exist"
Sep 07 10:47:41.983354 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.983336    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"]
Sep 07 10:47:41.987482 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:41.987463    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7596b5dfb4-79wwr"]
Sep 07 10:47:42.914676 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:42.914643    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:43.895310 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:43.895272    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbf9f45f-f69d-4f86-9a54-df8aca3f438d" path="/var/lib/kubelet/pods/fbf9f45f-f69d-4f86-9a54-df8aca3f438d/volumes"
Sep 07 10:47:48.895385 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:48.895349    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:55.873361 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:55.873323    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:47:55.888893 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:55.888864    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-mqmqd"]
Sep 07 10:47:58.875930 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:47:58.875898    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79bc45b668-llbcs_87887868-71be-4133-a616-8a0c3c55010d/authorino/0.log"
Sep 07 10:48:04.794996 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:04.794960    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-gvgn5_864c4745-6b6f-4b85-bd67-959a99ead247/copy-manifests/0.log"
Sep 07 10:48:04.801740 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:04.801716    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-gvgn5_864c4745-6b6f-4b85-bd67-959a99ead247/manager/0.log"
Sep 07 10:48:05.387935 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:05.387905    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b9564d484-4m8dq_5c4fb05f-510d-4d76-9410-a932d68305ee/maas-api/0.log"
Sep 07 10:48:05.612655 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:05.612623    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29812950-tmjhx_1c698d18-6ed4-4269-807a-8c503fbfbde2/cleanup/0.log"
Sep 07 10:48:05.716180 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:05.716110    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29812965-6t97r_d8f62a6b-4c91-4a25-a570-5f54bc98fae3/cleanup/0.log"
Sep 07 10:48:06.626160 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:06.626124    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79bc45b668-llbcs_87887868-71be-4133-a616-8a0c3c55010d/authorino/0.log"
Sep 07 10:48:06.962830 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:06.962746    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-g4kf6_86b2146f-98e9-4811-85ad-a4684ba61ed1/kuadrant-console-plugin/0.log"
Sep 07 10:48:07.200946 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:07.200911    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-mqmqd_1d95000e-1fb2-4b8a-9a2f-da48e0eefccc/limitador/0.log"
Sep 07 10:48:07.907004 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:07.906976    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7959f5887b-bgzv5_64dd96ce-65de-48bc-ab25-9d0e2e7e36a4/kube-auth-proxy/0.log"
Sep 07 10:48:09.223603 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:09.223571    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-szvgw_fe9f479b-fbcc-4127-bc1b-43467ee472db/main/0.log"
Sep 07 10:48:09.464792 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:09.464764    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-qbstv_fbb1aecf-5642-4506-8280-3e97e235ea48/main/0.log"
Sep 07 10:48:15.580448 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:15.580407    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:48:15.653131 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:15.653097    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:48:18.537620 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:18.537593    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jr9hg_bfd233c5-b9bd-47b1-aee1-1013a76983b5/global-pull-secret-syncer/0.log"
Sep 07 10:48:18.635276 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:18.635213    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log"
Sep 07 10:48:18.727790 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:18.727760    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-191.ec2.internal_c349833f5e025366a333ee7959fbf2b3/haproxy/0.log"
Sep 07 10:48:21.509110 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:21.509058    2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-sxqmk_969e1cf8-0598-41cd-a6d5-626d5fc54ea7/konnectivity-agent/0.log" limit=1024
Sep 07 10:48:23.209175 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:23.209122    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-79bc45b668-llbcs_87887868-71be-4133-a616-8a0c3c55010d/authorino/0.log"
Sep 07 10:48:23.297261 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:23.297233    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-g4kf6_86b2146f-98e9-4811-85ad-a4684ba61ed1/kuadrant-console-plugin/0.log"
Sep 07 10:48:23.365990 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:23.365959    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-mqmqd_1d95000e-1fb2-4b8a-9a2f-da48e0eefccc/limitador/0.log"
Sep 07 10:48:25.192966 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.192925    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px2dm_83a941cf-17ea-4e2e-890d-d898be6b5d0b/node-exporter/0.log"
Sep 07 10:48:25.213792 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.213770    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px2dm_83a941cf-17ea-4e2e-890d-d898be6b5d0b/kube-rbac-proxy/0.log"
Sep 07 10:48:25.234123 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.234101    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-px2dm_83a941cf-17ea-4e2e-890d-d898be6b5d0b/init-textfile/0.log"
Sep 07 10:48:25.417406 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.417368    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-g754t_790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea/kube-rbac-proxy-main/0.log"
Sep 07 10:48:25.438231 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.438205    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-g754t_790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea/kube-rbac-proxy-self/0.log"
Sep 07 10:48:25.460769 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.460736    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-g754t_790b4027-9f0b-4b46-9e4a-89c2ddf6f6ea/openshift-state-metrics/0.log"
Sep 07 10:48:25.498359 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.498331    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f8a3db7-9b27-4511-881b-a09e8ffa9b6e/prometheus/0.log"
Sep 07 10:48:25.518007 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.517983    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f8a3db7-9b27-4511-881b-a09e8ffa9b6e/config-reloader/0.log"
Sep 07 10:48:25.540756 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.540739    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f8a3db7-9b27-4511-881b-a09e8ffa9b6e/thanos-sidecar/0.log"
Sep 07 10:48:25.560926 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.560903    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f8a3db7-9b27-4511-881b-a09e8ffa9b6e/kube-rbac-proxy-web/0.log"
Sep 07 10:48:25.582284 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.582262    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f8a3db7-9b27-4511-881b-a09e8ffa9b6e/kube-rbac-proxy/0.log"
Sep 07 10:48:25.602560 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.602540    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f8a3db7-9b27-4511-881b-a09e8ffa9b6e/kube-rbac-proxy-thanos/0.log"
Sep 07 10:48:25.625940 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.625878    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f8a3db7-9b27-4511-881b-a09e8ffa9b6e/init-config-reloader/0.log"
Sep 07 10:48:25.696647 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.696624    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-8clrt_8f063994-2fa4-4b52-a4bf-3730feb3f383/prometheus-operator-admission-webhook/0.log"
Sep 07 10:48:25.810962 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.810936    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d856f4f5c-f98w6_330f5d33-f755-4067-8b32-86fd7d7cbf24/thanos-query/0.log"
Sep 07 10:48:25.836113 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.836088    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d856f4f5c-f98w6_330f5d33-f755-4067-8b32-86fd7d7cbf24/kube-rbac-proxy-web/0.log"
Sep 07 10:48:25.860547 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.860517    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d856f4f5c-f98w6_330f5d33-f755-4067-8b32-86fd7d7cbf24/kube-rbac-proxy/0.log"
Sep 07 10:48:25.880959 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.880897    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d856f4f5c-f98w6_330f5d33-f755-4067-8b32-86fd7d7cbf24/prom-label-proxy/0.log"
Sep 07 10:48:25.912943 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.912914    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d856f4f5c-f98w6_330f5d33-f755-4067-8b32-86fd7d7cbf24/kube-rbac-proxy-rules/0.log"
Sep 07 10:48:25.935981 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:25.935954    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-d856f4f5c-f98w6_330f5d33-f755-4067-8b32-86fd7d7cbf24/kube-rbac-proxy-metrics/0.log"
Sep 07 10:48:28.868811 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:28.868778    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-b47fc8985-fb67l_97751574-c5d9-4128-a817-5829988af14e/console/0.log"
Sep 07 10:48:29.069824 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.069784    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"]
Sep 07 10:48:29.070219 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.070203    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbf9f45f-f69d-4f86-9a54-df8aca3f438d" containerName="maas-api"
Sep 07 10:48:29.070323 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.070222    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbf9f45f-f69d-4f86-9a54-df8aca3f438d" containerName="maas-api"
Sep 07 10:48:29.070384 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.070336    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbf9f45f-f69d-4f86-9a54-df8aca3f438d" containerName="maas-api"
Sep 07 10:48:29.073384 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.073363    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.075998 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.075977    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhr67\"/\"kube-root-ca.crt\""
Sep 07 10:48:29.075998 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.075977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhr67\"/\"default-dockercfg-ftpfk\""
Sep 07 10:48:29.076943 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.076925    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhr67\"/\"openshift-service-ca.crt\""
Sep 07 10:48:29.083890 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.083869    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"]
Sep 07 10:48:29.219099 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.218992    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-proc\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.219099 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.219045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-sys\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.219301 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.219150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8z9\" (UniqueName: \"kubernetes.io/projected/69c41959-9bc9-4f3e-a146-fe64ef54a001-kube-api-access-4w8z9\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.219301 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.219187    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-lib-modules\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.219301 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.219270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-podres\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320478 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-podres\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-proc\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-sys\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320584    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8z9\" (UniqueName: \"kubernetes.io/projected/69c41959-9bc9-4f3e-a146-fe64ef54a001-kube-api-access-4w8z9\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-lib-modules\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320642    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-proc\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320681 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320642    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-podres\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320941 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320674    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-sys\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.320941 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.320774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69c41959-9bc9-4f3e-a146-fe64ef54a001-lib-modules\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.329519 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.329491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8z9\" (UniqueName: \"kubernetes.io/projected/69c41959-9bc9-4f3e-a146-fe64ef54a001-kube-api-access-4w8z9\") pod \"perf-node-gather-daemonset-jj6vd\" (UID: \"69c41959-9bc9-4f3e-a146-fe64ef54a001\") " pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.383324 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.383293    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:29.656618 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:29.656577    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"]
Sep 07 10:48:29.659600 ip-10-0-128-191 kubenswrapper[2560]: W0907 10:48:29.659574    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod69c41959_9bc9_4f3e_a146_fe64ef54a001.slice/crio-20ee8217146d4df2fd48c1110e44e602d350da490ce7a1aa25dfb0b6f45faad9 WatchSource:0}: Error finding container 20ee8217146d4df2fd48c1110e44e602d350da490ce7a1aa25dfb0b6f45faad9: Status 404 returned error can't find the container with id 20ee8217146d4df2fd48c1110e44e602d350da490ce7a1aa25dfb0b6f45faad9
Sep 07 10:48:30.191365 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:30.191327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd" event={"ID":"69c41959-9bc9-4f3e-a146-fe64ef54a001","Type":"ContainerStarted","Data":"1c19df9779c5f39453dbe6d90d171e5e40e40bf23280c0414cbaff573e67ca50"}
Sep 07 10:48:30.191802 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:30.191383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd" event={"ID":"69c41959-9bc9-4f3e-a146-fe64ef54a001","Type":"ContainerStarted","Data":"20ee8217146d4df2fd48c1110e44e602d350da490ce7a1aa25dfb0b6f45faad9"}
Sep 07 10:48:30.191802 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:30.191405    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:30.207302 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:30.207252    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd" podStartSLOduration=1.207236167 podStartE2EDuration="1.207236167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 10:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 10:48:30.207066186 +0000 UTC m=+2216.755581174" watchObservedRunningTime="2026-09-07 10:48:30.207236167 +0000 UTC m=+2216.755751153"
Sep 07 10:48:30.828281 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:30.828253    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nj8x8_4ab0827d-36bd-4e44-b152-b72cf5cf8861/dns/0.log"
Sep 07 10:48:30.848089 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:30.848045    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nj8x8_4ab0827d-36bd-4e44-b152-b72cf5cf8861/kube-rbac-proxy/0.log"
Sep 07 10:48:30.874375 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:30.874352    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cjhxw_679b04d9-df87-45a9-8bc1-8cb7d5c88758/dns-node-resolver/0.log"
Sep 07 10:48:31.448316 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:31.448283    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6587d68c88-wz6c8_3b6e72ba-6fa2-44ab-8d5f-8b943f196c1b/registry/0.log"
Sep 07 10:48:31.471280 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:31.471252    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-56glg_eb2119b6-8f00-4d3d-a2f8-3689940c55fd/node-ca/0.log"
Sep 07 10:48:33.000387 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:33.000360    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7959f5887b-bgzv5_64dd96ce-65de-48bc-ab25-9d0e2e7e36a4/kube-auth-proxy/0.log"
Sep 07 10:48:34.075474 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:34.075447    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jq7l9_d685669b-5eb2-4358-a543-f59a8df88f4e/serve-healthcheck-canary/0.log"
Sep 07 10:48:34.648788 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:34.648752    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6ft8s_89b831f6-f4b9-4ddf-baed-806b217f0c8e/kube-rbac-proxy/0.log"
Sep 07 10:48:34.672291 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:34.672250    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6ft8s_89b831f6-f4b9-4ddf-baed-806b217f0c8e/exporter/0.log"
Sep 07 10:48:34.693808 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:34.693775    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6ft8s_89b831f6-f4b9-4ddf-baed-806b217f0c8e/extractor/0.log"
Sep 07 10:48:36.205149 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:36.205123    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhr67/perf-node-gather-daemonset-jj6vd"
Sep 07 10:48:37.170653 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:37.170613    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-gvgn5_864c4745-6b6f-4b85-bd67-959a99ead247/manager/0.log"
Sep 07 10:48:37.196433 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:37.196408    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685c7f779d-gvgn5_864c4745-6b6f-4b85-bd67-959a99ead247/copy-manifests/0.log"
Sep 07 10:48:38.786804 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:38.786774    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c54b9f868-p2zsj_b3de56e5-6711-4537-b795-4660f5c8dde5/manager/0.log"
Sep 07 10:48:40.170227 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:40.170200    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7959f5887b-bgzv5_64dd96ce-65de-48bc-ab25-9d0e2e7e36a4/kube-auth-proxy/0.log"
Sep 07 10:48:42.418641 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:42.418608    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7959f5887b-bgzv5_64dd96ce-65de-48bc-ab25-9d0e2e7e36a4/kube-auth-proxy/0.log"
Sep 07 10:48:46.131955 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.131925    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shht6_1ce3557a-cc31-4782-9785-ed93485f273a/kube-multus-additional-cni-plugins/0.log"
Sep 07 10:48:46.155131 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.155100    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shht6_1ce3557a-cc31-4782-9785-ed93485f273a/egress-router-binary-copy/0.log"
Sep 07 10:48:46.176793 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.176761    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shht6_1ce3557a-cc31-4782-9785-ed93485f273a/cni-plugins/0.log"
Sep 07 10:48:46.197864 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.197826    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shht6_1ce3557a-cc31-4782-9785-ed93485f273a/bond-cni-plugin/0.log"
Sep 07 10:48:46.221376 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.221348    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shht6_1ce3557a-cc31-4782-9785-ed93485f273a/routeoverride-cni/0.log"
Sep 07 10:48:46.243509 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.243484    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shht6_1ce3557a-cc31-4782-9785-ed93485f273a/whereabouts-cni-bincopy/0.log"
Sep 07 10:48:46.269516 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.269486    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-shht6_1ce3557a-cc31-4782-9785-ed93485f273a/whereabouts-cni/0.log"
Sep 07 10:48:46.381835 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.381804    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ngwlm_c50d79e9-af4e-4927-b4f4-2cfd83f3290e/kube-multus/0.log"
Sep 07 10:48:46.445706 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.445627    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dbjml_232a763f-6ba8-4484-855b-ead84890b628/network-metrics-daemon/0.log"
Sep 07 10:48:46.460110 ip-10-0-128-191 kubenswrapper[2560]: I0907 10:48:46.460083    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dbjml_232a763f-6ba8-4484-855b-ead84890b628/kube-rbac-proxy/0.log"