Jul 24 22:46:22.068620 ip-10-0-136-221 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 22:46:22.429878 ip-10-0-136-221 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 22:46:22.429878 ip-10-0-136-221 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 22:46:22.429878 ip-10-0-136-221 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 22:46:22.429878 ip-10-0-136-221 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 22:46:22.430585 ip-10-0-136-221 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 22:46:22.430585 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.429955    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 22:46:22.435827 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435803    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 22:46:22.435827 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435825    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435831    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435836    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435846    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435850    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435853    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435856    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435859    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435862    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435865    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435868    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435871    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435873    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435876    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435879    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435883    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435887    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435890    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435892    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435895    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 22:46:22.435906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435898    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435900    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435902    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435906    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435908    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435911    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435914    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435917    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435920    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435922    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435925    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435927    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435930    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435933    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435935    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435938    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435941    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435943    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435946    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435955    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 22:46:22.436435 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435958    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435960    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435963    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435965    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435968    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435972    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435976    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435979    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435982    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435984    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435989    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435992    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435994    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.435999    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436002    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436004    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436011    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436013    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436016    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 22:46:22.437012 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436019    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436021    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436024    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436027    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436029    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436032    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436035    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436037    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436040    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436042    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436048    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436051    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436053    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436056    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436058    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436061    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436064    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436066    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436069    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 22:46:22.437762 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436071    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436074    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436076    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436079    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436081    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436084    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436086    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436089    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436092    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436095    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436098    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436100    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436103    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436106    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436108    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436111    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436113    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436116    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436118    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.436121    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 22:46:22.438297 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438200    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438217    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438221    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438227    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438232    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438236    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438239    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438242    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438245    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438247    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438250    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438265    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438268    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438271    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438274    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438276    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438280    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438283    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438285    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 22:46:22.438792 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438288    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438291    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438293    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438296    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438299    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438301    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438304    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438307    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438310    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438312    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438315    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438318    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438320    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438324    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438327    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438330    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438333    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438336    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438338    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438341    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 22:46:22.439373 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438344    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438346    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438349    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438351    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438353    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438356    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438358    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438361    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438363    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438366    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438370    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438383    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438388    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438392    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438395    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438398    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438401    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438403    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438406    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 22:46:22.439876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438409    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438411    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438414    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438417    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438419    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438422    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438424    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438427    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438431    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438435    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438438    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438441    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438443    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438446    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438448    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438451    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438454    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438456    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438459    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 22:46:22.440393 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438461    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438464    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438466    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438469    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438472    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438474    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438483    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438486    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438488    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438491    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438494    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438497    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438500    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438502    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438505    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438507    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438510    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438512    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438515    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438517    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 22:46:22.440865 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438521    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438523    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438954    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438959    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438962    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438965    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438967    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438971    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438973    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438977    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438979    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438982    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438985    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438987    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438990    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438993    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438996    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.438998    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439007    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439010    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 22:46:22.441386 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439012    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439015    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439018    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439020    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439023    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439028    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439031    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439035    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439038    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439040    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439043    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439045    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439048    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439051    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439054    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439057    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439060    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439063    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439065    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439068    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 22:46:22.441906 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439070    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439073    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439076    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439078    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439081    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439084    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439087    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439091    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439094    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439097    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439099    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439107    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439110    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439112    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439115    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439117    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439121    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439125    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439128    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 22:46:22.442414 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439130    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439133    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439136    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439138    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439141    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439145    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439148    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439151    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439153    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439156    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439159    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439161    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439164    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439166    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439169    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439171    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439174    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439176    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439179    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 22:46:22.442876 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439181    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439186    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439190    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439193    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439196    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439200    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439214    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439219    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439222    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439225    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439228    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439230    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439233    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439236    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439239    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439241    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439244    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439246    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439249    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439265    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 22:46:22.443360 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439268    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439271    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.439273    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439349    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439358    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439373    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439378    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439382    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439386    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439390    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439394    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439398    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439401    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439404    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439408    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439411    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439414    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439417    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439420    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439431    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439434    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439441    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439444    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439447    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 22:46:22.443847 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439450    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439454    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439458    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439461    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439464    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439468    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439471    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439474    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439477    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439481    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439484    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439488    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439491    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439494    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439497    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439500    2563 flags.go:64] FLAG: --enable-server="true"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439503    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439510    2563 flags.go:64] FLAG: --event-burst="100"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439514    2563 flags.go:64] FLAG: --event-qps="50"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439517    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439520    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439523    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439526    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439529    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439532    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 22:46:22.444453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439535    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439538    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439546    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439555    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439558    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439561    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439564    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439567    2563 flags.go:64] FLAG: --feature-gates=""
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439571    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439574    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439577    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439580    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439583    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439587    2563 flags.go:64] FLAG: --help="false"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439590    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439593    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439596    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439599    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439602    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439606    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439609    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439612    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439615    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 22:46:22.445051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439618    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439621    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439624    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439627    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439630    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439632    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439635    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439638    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439641    2563 flags.go:64] FLAG: --lock-file=""
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439644    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439647    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439651    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439657    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439666    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439669    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439672    2563 flags.go:64] FLAG: --logging-format="text"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439674    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439678    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439681    2563 flags.go:64] FLAG: --manifest-url=""
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439684    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439688    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439691    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439696    2563 flags.go:64] FLAG: --max-pods="110"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439699    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439702    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 22:46:22.445724 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439705    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439708    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439711    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439714    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439716    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439724    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439727    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439730    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439733    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439736    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439742    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439745    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439748    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439751    2563 flags.go:64] FLAG: --port="10250"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439754    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439757    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d6d07c63ddd3037c"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439760    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439763    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439767    2563 flags.go:64] FLAG: --register-node="true"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439770    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439774    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439783    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439786    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439789    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439793    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 22:46:22.446360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439796    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439799    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439802    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439805    2563 flags.go:64] FLAG: --runonce="false"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439808    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439813    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439816    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439819    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439822    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439826    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439829    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439832    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439835    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439838    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439841    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439844    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439847    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439850    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439853    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439858    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439861    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439864    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439871    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439874    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439877    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 22:46:22.446984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439881    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439884    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439887    2563 flags.go:64] FLAG: --v="2"
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439892    2563 flags.go:64] FLAG: --version="false"
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439903    2563 flags.go:64] FLAG: --vmodule=""
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439907    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.439911    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440052    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440056    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440059    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440063    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440065    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440069    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440072    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440075    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440077    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440080    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440083    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440085    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440087    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440090    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 22:46:22.447599 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440093    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440096    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440098    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440101    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440103    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440106    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440108    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440111    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440113    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440116    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440118    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440123    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440126    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440130    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440134    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440136    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440145    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440148    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440151    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440153    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 22:46:22.448111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440156    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440159    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440162    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440166    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440168    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440171    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440173    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440176    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440179    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440182    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440184    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440187    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440191    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440194    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440198    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440200    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440203    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440206    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440209    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440212    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 22:46:22.448614 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440214    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440217    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440220    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440223    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440226    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440228    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440231    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440234    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440236    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440245    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440248    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440267    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440270    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440273    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440275    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440280    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440282    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440285    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440288    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 22:46:22.449114 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440291    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440293    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440296    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440299    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440302    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440304    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440307    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440309    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440312    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440314    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440317    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440320    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440322    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440325    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440328    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440330    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440334    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440337    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440339    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440342    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 22:46:22.449611 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440344    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440348    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440352    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440356    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440359    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.440362    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.440855    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.447914    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.447936    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.447959    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.447966    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448031    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448037    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448040    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 22:46:22.450100 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448043    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448047    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448051    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448055    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448060    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448063    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448066    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448069    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448072    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448075    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448078    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448081    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448084    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448086    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448089    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448092    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448094    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448097    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448099    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448102    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 22:46:22.450472 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448104    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448107    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448110    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448113    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448115    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448118    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448122    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448126    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448129    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448131    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448134    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448137    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448140    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448143    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448146    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448150    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448154    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448157    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448160    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448163    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 22:46:22.450999 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448166    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448168    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448172    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448174    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448178    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448180    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448183    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448186    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448188    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448191    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448194    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448196    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448198    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448201    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448204    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448206    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448209    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448211    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448214    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448218    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 22:46:22.451507 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448220    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448224    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448227    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448229    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448232    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448235    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448237    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448240    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448242    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448245    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448248    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448268    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448273    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448276    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448278    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448281    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448285    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448288    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448290    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448293    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 22:46:22.451998 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448296    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448298    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448301    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448303    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448306    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448308    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448312    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448316    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448319    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448322    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448326    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448329    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448333    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448336    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448339    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 22:46:22.452508 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448342    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.448347    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448470    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448476    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448479    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448481    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448484    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448487    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448490    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448492    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448494    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448497    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448500    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448503    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448506    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 22:46:22.452873 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448508    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448511    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448514    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448516    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448519    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448521    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448524    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448527    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448531    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448535    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448539    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448542    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448544    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448547    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448550    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448554    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448557    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448560    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448563    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 22:46:22.453246 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448566    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448568    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448571    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448574    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448577    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448580    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448582    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448585    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448588    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448592    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448596    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448599    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448602    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448605    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448607    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448610    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448613    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448616    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448618    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 22:46:22.453725 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448621    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448624    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448627    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448629    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448632    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448634    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448637    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448640    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448642    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448645    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448648    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448651    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448654    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448656    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448659    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448661    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448664    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448666    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448669    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448672    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 22:46:22.454192 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448674    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448677    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448679    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448682    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448684    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448687    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448690    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448692    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448695    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448697    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448700    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448702    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448705    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448708    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448710    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448713    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448715    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448718    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448721    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448723    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448726    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 22:46:22.454687 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448729    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448731    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448734    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448737    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448739    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448742    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:22.448744    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.448749    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.448910    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.450950    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.452232    2563 server.go:1019] "Starting client certificate rotation"
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.452286    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.452333    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 22:46:22.455202 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.455138    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 22:46:22.458987 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.458959    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 22:46:22.466921 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.466896    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 24 22:46:22.469217 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.469196    2563 log.go:25] "Validated CRI v1 image API"
Jul 24 22:46:22.471369 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.471346    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 22:46:22.473668 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.473643    2563 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b2f998bf-0331-45e3-afe8-917e72b4ab7e:/dev/nvme0n1p4 ee53f228-de18-4923-9156-f83253f0dc32:/dev/nvme0n1p3]
Jul 24 22:46:22.473758 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.473665    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 22:46:22.476100 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.476077    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 22:46:22.479632 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.479504    2563 manager.go:216] Machine: {Timestamp:2026-07-24 22:46:22.478226915 +0000 UTC m=+0.301771528 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100713 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d3253848a8eb424deaa05d1fc3d88 SystemUUID:ec2d3253-848a-8eb4-24de-aa05d1fc3d88 BootID:9863d9b3-c936-4de0-81bc-423bb816a323 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5e:8a:0b:51:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5e:8a:0b:51:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:b4:c0:64:f8:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 22:46:22.479632 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.479622    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 22:46:22.479778 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.479765    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 22:46:22.482558 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.482522    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 22:46:22.482767 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.482562    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 22:46:22.483298 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.483285    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 22:46:22.483331 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.483300    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 22:46:22.483331 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.483308    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 22:46:22.483928 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.483916    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 22:46:22.484780 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.484768    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 22:46:22.486445 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.486430    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 22:46:22.486589 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.486579    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 22:46:22.486628 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.486615    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 22:46:22.486656 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.486635    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 22:46:22.486656 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.486651    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 22:46:22.486708 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.486661    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 24 22:46:22.486708 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.486672    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 22:46:22.487069 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.487057    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 22:46:22.487103 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.487077    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 22:46:22.488984 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.488962    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 22:46:22.490757 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.490742    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 22:46:22.490807 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.490774    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 22:46:22.492166 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492153    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 22:46:22.492214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492170    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 22:46:22.492214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492176    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 22:46:22.492214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492182    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 22:46:22.492214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492190    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 22:46:22.492214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492196    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 22:46:22.492214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492204    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 22:46:22.492214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492209    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 22:46:22.492214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492217    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 22:46:22.492438 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492223    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 22:46:22.492438 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492231    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 22:46:22.492438 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.492237    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 22:46:22.493628 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.493612    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 22:46:22.493669 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.493632    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 22:46:22.494396 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.494370    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 22:46:22.494464 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.494370    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 22:46:22.496220 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.496196    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 22:46:22.497557 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.497543    2563 server.go:1268] "Started kubelet"
Jul 24 22:46:22.497654 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.497628    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 22:46:22.497720 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.497686    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 22:46:22.497752 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.497741    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 22:46:22.501058 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.501024    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 22:46:22.501625 ip-10-0-136-221 systemd[1]: Started Kubernetes Kubelet.
Jul 24 22:46:22.502176 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.502150    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ngnn"
Jul 24 22:46:22.503752 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.503721    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 22:46:22.505656 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.505633    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 22:46:22.505957 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.505941    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ngnn"
Jul 24 22:46:22.508156 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.507130    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-221.ec2.internal.18c55b45c2924adb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-221.ec2.internal,UID:ip-10-0-136-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-221.ec2.internal,},FirstTimestamp:2026-07-24 22:46:22.497508059 +0000 UTC m=+0.321052670,LastTimestamp:2026-07-24 22:46:22.497508059 +0000 UTC m=+0.321052670,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-221.ec2.internal,}"
Jul 24 22:46:22.509635 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.509614    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 22:46:22.509635 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.509616    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 22:46:22.509851 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.509738    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:22.510292 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.510225    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 22:46:22.510292 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.510229    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 22:46:22.510292 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.510249    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 22:46:22.510449 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.510418    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 22:46:22.510449 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.510428    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 22:46:22.510885 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.510867    2563 factory.go:55] Registering systemd factory
Jul 24 22:46:22.510885 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.510887    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 24 22:46:22.511112 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.511100    2563 factory.go:153] Registering CRI-O factory
Jul 24 22:46:22.511151 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.511115    2563 factory.go:223] Registration of the crio container factory successfully
Jul 24 22:46:22.511182 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.511161    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 22:46:22.511225 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.511182    2563 factory.go:103] Registering Raw factory
Jul 24 22:46:22.511225 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.511194    2563 manager.go:1196] Started watching for new ooms in manager
Jul 24 22:46:22.511586 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.511570    2563 manager.go:349] Starting recovery of all containers
Jul 24 22:46:22.512767 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.512658    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 22:46:22.515761 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.515720    2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-221.ec2.internal\" not found" node="ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.523607 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.523581    2563 manager.go:354] Recovery completed
Jul 24 22:46:22.528236 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.528212    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 22:46:22.530328 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.530245    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 22:46:22.530428 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.530343    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 22:46:22.530428 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.530354    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientPID"
Jul 24 22:46:22.530883 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.530868    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 22:46:22.530883 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.530882    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 22:46:22.530964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.530899    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 22:46:22.530964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.530941    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 22:46:22.532186 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.532165    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 22:46:22.532283 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.532207    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 22:46:22.532283 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.532224    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientPID"
Jul 24 22:46:22.532340 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.532312    2563 policy_none.go:49] "None policy: Start"
Jul 24 22:46:22.532340 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.532327    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 22:46:22.532340 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.532337    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 22:46:22.533167 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.533157    2563 policy_none.go:47] "Start"
Jul 24 22:46:22.566127 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.565936    2563 manager.go:337] "Starting Device Plugin manager"
Jul 24 22:46:22.566303 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.566146    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 22:46:22.566303 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.566158    2563 server.go:87] "Starting device plugin registration server"
Jul 24 22:46:22.566537 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.566524    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 22:46:22.566575 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.566541    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 22:46:22.566698 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.566674    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 22:46:22.566798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.566774    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 22:46:22.566798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.566783    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 22:46:22.567306 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.567287    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 22:46:22.567384 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.567343    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:22.643426 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.643377    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 22:46:22.644642 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.644624    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 22:46:22.644705 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.644669    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 22:46:22.644705 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.644691    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 22:46:22.644801 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.644732    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 22:46:22.647401 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.647377    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 22:46:22.667421 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.667394    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 22:46:22.669167 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.669144    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 22:46:22.669321 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.669177    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 22:46:22.669321 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.669188    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientPID"
Jul 24 22:46:22.669321 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.669212    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.677221 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.677199    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.677319 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.677229    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-221.ec2.internal\": node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:22.690663 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.690584    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:22.746450 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.745777    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal"]
Jul 24 22:46:22.746565 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.746509    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 22:46:22.747592 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.747574    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 22:46:22.747679 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.747602    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 22:46:22.747679 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.747613    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientPID"
Jul 24 22:46:22.748744 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.748729    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 22:46:22.748846 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.748833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.748879 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.748862    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 22:46:22.749626 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.749612    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 22:46:22.749697 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.749612    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 22:46:22.749697 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.749635    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 22:46:22.749697 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.749645    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientPID"
Jul 24 22:46:22.749697 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.749652    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 22:46:22.749697 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.749666    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientPID"
Jul 24 22:46:22.750804 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.750786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.750904 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.750817    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 22:46:22.751596 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.751583    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 22:46:22.751678 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.751611    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 22:46:22.751678 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.751628    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeHasSufficientPID"
Jul 24 22:46:22.768993 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.768966    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-221.ec2.internal\" not found" node="ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.772747 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.772728    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-221.ec2.internal\" not found" node="ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.791112 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.791086    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:22.891917 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.891869    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:22.912313 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.912287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29ce5794fceb53a7c97a5acddf19c77a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal\" (UID: \"29ce5794fceb53a7c97a5acddf19c77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.912399 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.912319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29ce5794fceb53a7c97a5acddf19c77a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal\" (UID: \"29ce5794fceb53a7c97a5acddf19c77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.912399 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:22.912341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b24225b2a0a08b20d14ab2b1793c0b0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-221.ec2.internal\" (UID: \"5b24225b2a0a08b20d14ab2b1793c0b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:22.992274 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:22.992189    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:23.012588 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.012551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29ce5794fceb53a7c97a5acddf19c77a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal\" (UID: \"29ce5794fceb53a7c97a5acddf19c77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.012705 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.012597    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29ce5794fceb53a7c97a5acddf19c77a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal\" (UID: \"29ce5794fceb53a7c97a5acddf19c77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.012705 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.012659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29ce5794fceb53a7c97a5acddf19c77a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal\" (UID: \"29ce5794fceb53a7c97a5acddf19c77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.012705 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.012682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b24225b2a0a08b20d14ab2b1793c0b0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-221.ec2.internal\" (UID: \"5b24225b2a0a08b20d14ab2b1793c0b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.012705 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.012660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29ce5794fceb53a7c97a5acddf19c77a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal\" (UID: \"29ce5794fceb53a7c97a5acddf19c77a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.012851 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.012722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b24225b2a0a08b20d14ab2b1793c0b0-config\") pod \"kube-apiserver-proxy-ip-10-0-136-221.ec2.internal\" (UID: \"5b24225b2a0a08b20d14ab2b1793c0b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.071749 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.071705    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.075597 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.075505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.092527 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:23.092485    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:23.192979 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:23.192943    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:23.293599 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:23.293517    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:23.394080 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:23.394037    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:23.452644 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.452605    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 22:46:23.453132 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.452767    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 22:46:23.453132 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.452792    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 22:46:23.495160 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:23.495129    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:23.509693 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.509649    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 22:41:23 +0000 UTC" deadline="2028-05-02 14:37:03.749990308 +0000 UTC"
Jul 24 22:46:23.509693 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.509682    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15543h50m40.240312451s"
Jul 24 22:46:23.595896 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:23.595862    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-221.ec2.internal\" not found"
Jul 24 22:46:23.655631 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:23.655590    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ce5794fceb53a7c97a5acddf19c77a.slice/crio-c558966bd745f61173555db39592d7f16f76c5cba46bf596c110a5ca2e6bf29c WatchSource:0}: Error finding container c558966bd745f61173555db39592d7f16f76c5cba46bf596c110a5ca2e6bf29c: Status 404 returned error can't find the container with id c558966bd745f61173555db39592d7f16f76c5cba46bf596c110a5ca2e6bf29c
Jul 24 22:46:23.660501 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.660477    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:46:23.681473 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.681446    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 22:46:23.693509 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:23.693484    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b24225b2a0a08b20d14ab2b1793c0b0.slice/crio-170bd4480f9822ad8c597ccb5b9222cedf56f00e25c3bcfd620ce5386ea9ae3f WatchSource:0}: Error finding container 170bd4480f9822ad8c597ccb5b9222cedf56f00e25c3bcfd620ce5386ea9ae3f: Status 404 returned error can't find the container with id 170bd4480f9822ad8c597ccb5b9222cedf56f00e25c3bcfd620ce5386ea9ae3f
Jul 24 22:46:23.710727 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.710686    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.723600 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.723562    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 22:46:23.725177 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.725158    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal"
Jul 24 22:46:23.731883 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.731861    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 22:46:23.734297 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:23.734276    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 22:46:25.678101 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.678070    2563 apiserver.go:52] "Watching apiserver"
Jul 24 22:46:25.686182 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.686151    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 22:46:25.689536 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.689475    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-nrlfg","openshift-image-registry/node-ca-8qv9m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal","openshift-multus/multus-2gwg8","openshift-multus/network-metrics-daemon-nspfm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b","openshift-dns/node-resolver-6lphx","openshift-multus/multus-additional-cni-plugins-5bf8x","openshift-network-diagnostics/network-check-target-48cpz","openshift-network-operator/iptables-alerter-cfv6k","openshift-ovn-kubernetes/ovnkube-node-sqlz6","kube-system/global-pull-secret-syncer-rv5zf","kube-system/konnectivity-agent-85c4k","kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal"]
Jul 24 22:46:25.691457 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.691431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.693118 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.693081    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:25.693868 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.693849    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f6mpq\""
Jul 24 22:46:25.694107 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.694093    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 22:46:25.694169 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.694124    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 22:46:25.694915 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.694882    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.695570 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.695549    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c457g\""
Jul 24 22:46:25.695676 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.695580    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 22:46:25.695841 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.695809    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 22:46:25.696912 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.696882    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.697621 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.697441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 22:46:25.697621 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.697507    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 22:46:25.697849 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.697833    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 22:46:25.697934 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.697855    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 22:46:25.698063 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.698047    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 22:46:25.698190 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.698132    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k8kk5\""
Jul 24 22:46:25.698838 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.698821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:25.698918 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.698892    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:25.699193 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.699173    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 22:46:25.699436 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.699411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lgl47\""
Jul 24 22:46:25.700071 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.700050    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 22:46:25.702513 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.702492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.704511 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.704492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:25.704629 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.704582    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:25.704887 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.704866    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 22:46:25.705169 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.705147    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 22:46:25.705255 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.705199    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 22:46:25.705255 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.705150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pt9d6\""
Jul 24 22:46:25.706627 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.706609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.708791 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.708773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.709202 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.709166    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 22:46:25.709911 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.709828    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 22:46:25.709911 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.709904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xn59k\""
Jul 24 22:46:25.710862 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.710845    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.710957 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.710890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31ca8850-9999-4ca9-be05-4dc4f733d845-agent-certs\") pod \"konnectivity-agent-85c4k\" (UID: \"31ca8850-9999-4ca9-be05-4dc4f733d845\") " pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:25.710957 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.710916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-socket-dir-parent\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.710957 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.710948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-daemon-config\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711101 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-multus-certs\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711101 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slj4l\" (UniqueName: \"kubernetes.io/projected/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-kube-api-access-slj4l\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711197 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-registration-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.711197 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-device-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.711292 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-cni-binary-copy\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711292 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-cni-bin\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711386 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-os-release\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.711386 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.711386 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cf7r\" (UniqueName: \"kubernetes.io/projected/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-kube-api-access-8cf7r\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.711386 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7g2\" (UniqueName: \"kubernetes.io/projected/06426582-6029-4f0b-a88f-eea7b0a829d6-kube-api-access-rq7g2\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.711583 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31ca8850-9999-4ca9-be05-4dc4f733d845-konnectivity-ca\") pod \"konnectivity-agent-85c4k\" (UID: \"31ca8850-9999-4ca9-be05-4dc4f733d845\") " pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:25.711583 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711457    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-cnibin\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711583 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711481    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 22:46:25.711583 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711543    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xf75t\""
Jul 24 22:46:25.711583 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-k8s-cni-cncf-io\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-conf-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711617    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711546    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-socket-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-system-cni-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-os-release\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.711763 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2wz4\" (UniqueName: \"kubernetes.io/projected/35f71564-7bff-4960-a63b-584726e054eb-kube-api-access-f2wz4\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-sys-fs\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-hostroot\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-system-cni-dir\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.711973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-cni-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06426582-6029-4f0b-a88f-eea7b0a829d6-tmp-dir\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712039    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595qg\" (UniqueName: \"kubernetes.io/projected/cd192207-7476-4816-a50d-ec3a230d5d73-kube-api-access-595qg\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.712146 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712106    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-netns\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.712539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-cni-multus\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.712539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:25.712539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.712539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06426582-6029-4f0b-a88f-eea7b0a829d6-hosts-file\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.712539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-cnibin\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.712539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-kubelet\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.712539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.712383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-etc-kubernetes\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.713325 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.713302    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 22:46:25.713325 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.713317    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 22:46:25.713616 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.713542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 22:46:25.713698 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.713674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xxz4q\""
Jul 24 22:46:25.714016 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.713996    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.714848 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.714826    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 22:46:25.716339 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.716318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 22:46:25.716765 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.716747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.716877 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.716846    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:25.717053 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.716877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 22:46:25.717053 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.716896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6llp\""
Jul 24 22:46:25.717482 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.717381    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 22:46:25.718481 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.717717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 22:46:25.718481 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.717881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 22:46:25.719973 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.719880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 22:46:25.731731 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.731706    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f7fzr"
Jul 24 22:46:25.741723 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.741701    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f7fzr"
Jul 24 22:46:25.801649 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.801611    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 22:46:25.813223 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e316233-25e3-4089-9d90-006bf80dbc8f-host-slash\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.813223 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.813466 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-socket-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.813466 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.813466 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysconfig\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.813466 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-host\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.813466 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813395    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-system-cni-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.813466 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2wz4\" (UniqueName: \"kubernetes.io/projected/35f71564-7bff-4960-a63b-584726e054eb-kube-api-access-f2wz4\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:25.813466 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813401    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-socket-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.813466 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-sys-fs\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/303ae2ca-2b38-4094-9507-21820dc52eeb-kubelet-config\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813479    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-etc-selinux\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-system-cni-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813496    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-kubelet\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-systemd\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813594    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-sys-fs\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-node-log\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-systemd\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-hostroot\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813727    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-system-cni-dir\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813764    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-run-netns\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.813825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-hostroot\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813858    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-system-cni-dir\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813922    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-cni-multus\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.813991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-cni-multus\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysctl-conf\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a332a24-918a-4476-b1d4-a013ede37685-etc-tuned\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-ovn\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814159    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-kubelet\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.814182    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-env-overrides\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-socket-dir-parent\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-multus-certs\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-kubelet\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.814400 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slj4l\" (UniqueName: \"kubernetes.io/projected/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-kube-api-access-slj4l\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-registration-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-device-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e316233-25e3-4089-9d90-006bf80dbc8f-iptables-alerter-script\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-cni-binary-copy\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.814392    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs podName:35f71564-7bff-4960-a63b-584726e054eb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:26.314346903 +0000 UTC m=+2.947625827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs") pod "network-metrics-daemon-nspfm" (UID: "35f71564-7bff-4960-a63b-584726e054eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-systemd-units\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-registration-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-multus-certs\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-cni-bin\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-socket-dir-parent\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-os-release\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-device-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814541    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-var-lib-cni-bin\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cf7r\" (UniqueName: \"kubernetes.io/projected/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-kube-api-access-8cf7r\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-os-release\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.815145 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-modprobe-d\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysctl-d\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovnkube-config\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovn-node-metrics-cert\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7g2\" (UniqueName: \"kubernetes.io/projected/06426582-6029-4f0b-a88f-eea7b0a829d6-kube-api-access-rq7g2\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-k8s-cni-cncf-io\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1746104c-a257-4d75-ba28-02305cbb01ba-host\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-slash\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-k8s-cni-cncf-io\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814816    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovnkube-script-lib\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b49md\" (UniqueName: \"kubernetes.io/projected/9a332a24-918a-4476-b1d4-a013ede37685-kube-api-access-b49md\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wx6d\" (UniqueName: \"kubernetes.io/projected/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-kube-api-access-4wx6d\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-os-release\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-lib-modules\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1746104c-a257-4d75-ba28-02305cbb01ba-serviceca\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-etc-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-os-release\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.815798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-cni-netd\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.814979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815005    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-cni-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/303ae2ca-2b38-4094-9507-21820dc52eeb-dbus\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815042    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-var-lib-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06426582-6029-4f0b-a88f-eea7b0a829d6-tmp-dir\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595qg\" (UniqueName: \"kubernetes.io/projected/cd192207-7476-4816-a50d-ec3a230d5d73-kube-api-access-595qg\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-cni-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-netns\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815190    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-host-run-netns\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a332a24-918a-4476-b1d4-a013ede37685-tmp\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p46x4\" (UniqueName: \"kubernetes.io/projected/1746104c-a257-4d75-ba28-02305cbb01ba-kube-api-access-p46x4\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06426582-6029-4f0b-a88f-eea7b0a829d6-tmp-dir\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7jh\" (UniqueName: \"kubernetes.io/projected/7e316233-25e3-4089-9d90-006bf80dbc8f-kube-api-access-bs7jh\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.816515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06426582-6029-4f0b-a88f-eea7b0a829d6-hosts-file\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-cnibin\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-etc-kubernetes\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cd192207-7476-4816-a50d-ec3a230d5d73-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-var-lib-kubelet\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06426582-6029-4f0b-a88f-eea7b0a829d6-hosts-file\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-cnibin\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-etc-kubernetes\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31ca8850-9999-4ca9-be05-4dc4f733d845-agent-certs\") pod \"konnectivity-agent-85c4k\" (UID: \"31ca8850-9999-4ca9-be05-4dc4f733d845\") " pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-daemon-config\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-kubernetes\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-log-socket\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-cni-bin\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-cni-binary-copy\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815783    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31ca8850-9999-4ca9-be05-4dc4f733d845-konnectivity-ca\") pod \"konnectivity-agent-85c4k\" (UID: \"31ca8850-9999-4ca9-be05-4dc4f733d845\") " pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:25.817165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-cnibin\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-conf-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815934    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-run\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.815980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-sys\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.816017    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.816101    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-conf-dir\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.816182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-cnibin\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.816213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd192207-7476-4816-a50d-ec3a230d5d73-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.816482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-cni-binary-copy\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.816695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/31ca8850-9999-4ca9-be05-4dc4f733d845-konnectivity-ca\") pod \"konnectivity-agent-85c4k\" (UID: \"31ca8850-9999-4ca9-be05-4dc4f733d845\") " pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:25.817812 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.816697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-multus-daemon-config\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.819594 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.819570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/31ca8850-9999-4ca9-be05-4dc4f733d845-agent-certs\") pod \"konnectivity-agent-85c4k\" (UID: \"31ca8850-9999-4ca9-be05-4dc4f733d845\") " pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:25.828892 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.828336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slj4l\" (UniqueName: \"kubernetes.io/projected/f8634fe9-3c65-480b-ab1c-ab306e77b9f6-kube-api-access-slj4l\") pod \"multus-2gwg8\" (UID: \"f8634fe9-3c65-480b-ab1c-ab306e77b9f6\") " pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:25.828892 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.828835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7g2\" (UniqueName: \"kubernetes.io/projected/06426582-6029-4f0b-a88f-eea7b0a829d6-kube-api-access-rq7g2\") pod \"node-resolver-6lphx\" (UID: \"06426582-6029-4f0b-a88f-eea7b0a829d6\") " pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:25.829252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.829214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-595qg\" (UniqueName: \"kubernetes.io/projected/cd192207-7476-4816-a50d-ec3a230d5d73-kube-api-access-595qg\") pod \"multus-additional-cni-plugins-5bf8x\" (UID: \"cd192207-7476-4816-a50d-ec3a230d5d73\") " pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:25.829461 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.829441    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cf7r\" (UniqueName: \"kubernetes.io/projected/a1f3294d-e87d-4096-a9a9-0e943bd9b98a-kube-api-access-8cf7r\") pod \"aws-ebs-csi-driver-node-f446b\" (UID: \"a1f3294d-e87d-4096-a9a9-0e943bd9b98a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:25.829550 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.829511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2wz4\" (UniqueName: \"kubernetes.io/projected/35f71564-7bff-4960-a63b-584726e054eb-kube-api-access-f2wz4\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:25.842732 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.842671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal" event={"ID":"5b24225b2a0a08b20d14ab2b1793c0b0","Type":"ContainerStarted","Data":"170bd4480f9822ad8c597ccb5b9222cedf56f00e25c3bcfd620ce5386ea9ae3f"}
Jul 24 22:46:25.843762 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.843738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal" event={"ID":"29ce5794fceb53a7c97a5acddf19c77a","Type":"ContainerStarted","Data":"c558966bd745f61173555db39592d7f16f76c5cba46bf596c110a5ca2e6bf29c"}
Jul 24 22:46:25.916813 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-var-lib-kubelet\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.916813 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-kubernetes\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-log-socket\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-cni-bin\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-var-lib-kubelet\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-kubernetes\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-log-socket\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-cni-bin\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.916993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917035    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-run\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917072 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-sys\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e316233-25e3-4089-9d90-006bf80dbc8f-host-slash\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.917088    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-run\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysconfig\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-host\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-sys\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.917166    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret podName:303ae2ca-2b38-4094-9507-21820dc52eeb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:26.417143217 +0000 UTC m=+3.050422142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret") pod "global-pull-secret-syncer-rv5zf" (UID: "303ae2ca-2b38-4094-9507-21820dc52eeb") : object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysconfig\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e316233-25e3-4089-9d90-006bf80dbc8f-host-slash\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/303ae2ca-2b38-4094-9507-21820dc52eeb-kubelet-config\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-host\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-kubelet\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-systemd\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/303ae2ca-2b38-4094-9507-21820dc52eeb-kubelet-config\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-node-log\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.917625 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-kubelet\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-systemd\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917309    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-systemd\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-run-netns\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-systemd\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-node-log\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysctl-conf\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-run-netns\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a332a24-918a-4476-b1d4-a013ede37685-etc-tuned\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-ovn\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-env-overrides\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917499    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e316233-25e3-4089-9d90-006bf80dbc8f-iptables-alerter-script\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-systemd-units\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysctl-conf\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.918422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917579    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-modprobe-d\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-run-ovn\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysctl-d\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-systemd-units\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovnkube-config\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovn-node-metrics-cert\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1746104c-a257-4d75-ba28-02305cbb01ba-host\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-slash\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-modprobe-d\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovnkube-script-lib\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49md\" (UniqueName: \"kubernetes.io/projected/9a332a24-918a-4476-b1d4-a013ede37685-kube-api-access-b49md\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-slash\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wx6d\" (UniqueName: \"kubernetes.io/projected/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-kube-api-access-4wx6d\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-lib-modules\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1746104c-a257-4d75-ba28-02305cbb01ba-host\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-etc-sysctl-d\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1746104c-a257-4d75-ba28-02305cbb01ba-serviceca\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.917988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-env-overrides\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919252 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918006    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-etc-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918017    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a332a24-918a-4476-b1d4-a013ede37685-lib-modules\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-cni-netd\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/303ae2ca-2b38-4094-9507-21820dc52eeb-dbus\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918147    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-var-lib-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a332a24-918a-4476-b1d4-a013ede37685-tmp\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p46x4\" (UniqueName: \"kubernetes.io/projected/1746104c-a257-4d75-ba28-02305cbb01ba-kube-api-access-p46x4\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7jh\" (UniqueName: \"kubernetes.io/projected/7e316233-25e3-4089-9d90-006bf80dbc8f-kube-api-access-bs7jh\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovnkube-config\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1746104c-a257-4d75-ba28-02305cbb01ba-serviceca\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovnkube-script-lib\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-etc-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/303ae2ca-2b38-4094-9507-21820dc52eeb-dbus\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e316233-25e3-4089-9d90-006bf80dbc8f-iptables-alerter-script\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-host-cni-netd\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.919930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.918740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-var-lib-openvswitch\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.920465 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.920021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a332a24-918a-4476-b1d4-a013ede37685-etc-tuned\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.920850 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.920829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-ovn-node-metrics-cert\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.921010 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.920994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a332a24-918a-4476-b1d4-a013ede37685-tmp\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.924814 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.924786    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 22:46:25.924814 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.924816    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 22:46:25.924994 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.924829    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tts6v for pod openshift-network-diagnostics/network-check-target-48cpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:25.924994 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:25.924896    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v podName:7fb93a54-2070-48f9-9810-e3f4acb7250c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:26.424876883 +0000 UTC m=+3.058155820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tts6v" (UniqueName: "kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v") pod "network-check-target-48cpz" (UID: "7fb93a54-2070-48f9-9810-e3f4acb7250c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:25.927351 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.927317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49md\" (UniqueName: \"kubernetes.io/projected/9a332a24-918a-4476-b1d4-a013ede37685-kube-api-access-b49md\") pod \"tuned-nrlfg\" (UID: \"9a332a24-918a-4476-b1d4-a013ede37685\") " pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:25.927444 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.927357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p46x4\" (UniqueName: \"kubernetes.io/projected/1746104c-a257-4d75-ba28-02305cbb01ba-kube-api-access-p46x4\") pod \"node-ca-8qv9m\" (UID: \"1746104c-a257-4d75-ba28-02305cbb01ba\") " pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:25.927771 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.927749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wx6d\" (UniqueName: \"kubernetes.io/projected/4d27a0bc-41c6-480e-a4e9-9c0386da97f0-kube-api-access-4wx6d\") pod \"ovnkube-node-sqlz6\" (UID: \"4d27a0bc-41c6-480e-a4e9-9c0386da97f0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:25.927853 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:25.927795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7jh\" (UniqueName: \"kubernetes.io/projected/7e316233-25e3-4089-9d90-006bf80dbc8f-kube-api-access-bs7jh\") pod \"iptables-alerter-cfv6k\" (UID: \"7e316233-25e3-4089-9d90-006bf80dbc8f\") " pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:26.004585 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.004487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6lphx"
Jul 24 22:46:26.012610 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.012572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:26.023557 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.023530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5bf8x"
Jul 24 22:46:26.028563 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.028542    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2gwg8"
Jul 24 22:46:26.036544 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.036498    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b"
Jul 24 22:46:26.043594 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.043569    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nrlfg"
Jul 24 22:46:26.059093 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.059065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8qv9m"
Jul 24 22:46:26.068293 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.068272    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cfv6k"
Jul 24 22:46:26.073478 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.073459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:26.118839 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.118809    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 22:46:26.124289 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.124260    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 22:46:26.322444 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.322358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:26.322604 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.322506    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:26.322604 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.322595    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs podName:35f71564-7bff-4960-a63b-584726e054eb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:27.322577949 +0000 UTC m=+3.955856869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs") pod "network-metrics-daemon-nspfm" (UID: "35f71564-7bff-4960-a63b-584726e054eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:26.423624 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.423582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:26.423779 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.423736    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:26.423835 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.423807    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret podName:303ae2ca-2b38-4094-9507-21820dc52eeb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:27.42378952 +0000 UTC m=+4.057068441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret") pod "global-pull-secret-syncer-rv5zf" (UID: "303ae2ca-2b38-4094-9507-21820dc52eeb") : object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:26.524309 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.524266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:26.524483 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.524434    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 22:46:26.524483 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.524455    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 22:46:26.524483 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.524465    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tts6v for pod openshift-network-diagnostics/network-check-target-48cpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:26.524635 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.524541    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v podName:7fb93a54-2070-48f9-9810-e3f4acb7250c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:27.524508216 +0000 UTC m=+4.157787156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tts6v" (UniqueName: "kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v") pod "network-check-target-48cpz" (UID: "7fb93a54-2070-48f9-9810-e3f4acb7250c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:26.679258 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.679232    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06426582_6029_4f0b_a88f_eea7b0a829d6.slice/crio-d54df8fbc5f803e472158b82efd596f58dd95e7a38f26396a602de7e03647989 WatchSource:0}: Error finding container d54df8fbc5f803e472158b82efd596f58dd95e7a38f26396a602de7e03647989: Status 404 returned error can't find the container with id d54df8fbc5f803e472158b82efd596f58dd95e7a38f26396a602de7e03647989
Jul 24 22:46:26.680385 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.680359    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f3294d_e87d_4096_a9a9_0e943bd9b98a.slice/crio-232b905769016111947f6736f2cda72aa42b5ad63669e2cf42dd0bca70198796 WatchSource:0}: Error finding container 232b905769016111947f6736f2cda72aa42b5ad63669e2cf42dd0bca70198796: Status 404 returned error can't find the container with id 232b905769016111947f6736f2cda72aa42b5ad63669e2cf42dd0bca70198796
Jul 24 22:46:26.682450 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.682342    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e316233_25e3_4089_9d90_006bf80dbc8f.slice/crio-fad6a3229d4a4b55f3140a7497799a2a646d5d47bcbce374334c05903f7326ec WatchSource:0}: Error finding container fad6a3229d4a4b55f3140a7497799a2a646d5d47bcbce374334c05903f7326ec: Status 404 returned error can't find the container with id fad6a3229d4a4b55f3140a7497799a2a646d5d47bcbce374334c05903f7326ec
Jul 24 22:46:26.685376 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.685352    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8634fe9_3c65_480b_ab1c_ab306e77b9f6.slice/crio-06ff3daf8718b37dce6293b609c3c560f7e39d7363acdb40607a3898b488cd9d WatchSource:0}: Error finding container 06ff3daf8718b37dce6293b609c3c560f7e39d7363acdb40607a3898b488cd9d: Status 404 returned error can't find the container with id 06ff3daf8718b37dce6293b609c3c560f7e39d7363acdb40607a3898b488cd9d
Jul 24 22:46:26.687252 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.687225    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a332a24_918a_4476_b1d4_a013ede37685.slice/crio-ff3550a0607cf6ba45ec2769aa0065a20b14b2f7db410a57e5ae5d675b150736 WatchSource:0}: Error finding container ff3550a0607cf6ba45ec2769aa0065a20b14b2f7db410a57e5ae5d675b150736: Status 404 returned error can't find the container with id ff3550a0607cf6ba45ec2769aa0065a20b14b2f7db410a57e5ae5d675b150736
Jul 24 22:46:26.688263 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.688237    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d27a0bc_41c6_480e_a4e9_9c0386da97f0.slice/crio-70ef009a36b92bc781176538dc2d95ad376cd47b791fc68b43dac103992fe6f1 WatchSource:0}: Error finding container 70ef009a36b92bc781176538dc2d95ad376cd47b791fc68b43dac103992fe6f1: Status 404 returned error can't find the container with id 70ef009a36b92bc781176538dc2d95ad376cd47b791fc68b43dac103992fe6f1
Jul 24 22:46:26.689237 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.689200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ca8850_9999_4ca9_be05_4dc4f733d845.slice/crio-bbc9f6f03fe4ea21dcaef3f4c5ea12166dc769e0999e447a7dfa48407babf486 WatchSource:0}: Error finding container bbc9f6f03fe4ea21dcaef3f4c5ea12166dc769e0999e447a7dfa48407babf486: Status 404 returned error can't find the container with id bbc9f6f03fe4ea21dcaef3f4c5ea12166dc769e0999e447a7dfa48407babf486
Jul 24 22:46:26.690452 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.690265    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1746104c_a257_4d75_ba28_02305cbb01ba.slice/crio-e3b2529d3c34dbbc3eb50604f2be7ecddebd88aa44185f8db276fd4ef0676b0b WatchSource:0}: Error finding container e3b2529d3c34dbbc3eb50604f2be7ecddebd88aa44185f8db276fd4ef0676b0b: Status 404 returned error can't find the container with id e3b2529d3c34dbbc3eb50604f2be7ecddebd88aa44185f8db276fd4ef0676b0b
Jul 24 22:46:26.691886 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:46:26.691861    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd192207_7476_4816_a50d_ec3a230d5d73.slice/crio-c17fe150d9351acc8ad813c05ad36ec8529f3845d18b21e8e8795bc2cd75febf WatchSource:0}: Error finding container c17fe150d9351acc8ad813c05ad36ec8529f3845d18b21e8e8795bc2cd75febf: Status 404 returned error can't find the container with id c17fe150d9351acc8ad813c05ad36ec8529f3845d18b21e8e8795bc2cd75febf
Jul 24 22:46:26.744585 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.744550    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:41:25 +0000 UTC" deadline="2028-02-02 17:24:56.492506796 +0000 UTC"
Jul 24 22:46:26.744585 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.744579    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13386h38m29.747930507s"
Jul 24 22:46:26.836302 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.836085    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:26.836490 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:26.836422    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:26.846986 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.846940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal" event={"ID":"5b24225b2a0a08b20d14ab2b1793c0b0","Type":"ContainerStarted","Data":"9ea98740f60675c6c99070efc58e14ae2658662bfbc60a173db2e711cb97e8fd"}
Jul 24 22:46:26.848263 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.848239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrlfg" event={"ID":"9a332a24-918a-4476-b1d4-a013ede37685","Type":"ContainerStarted","Data":"ff3550a0607cf6ba45ec2769aa0065a20b14b2f7db410a57e5ae5d675b150736"}
Jul 24 22:46:26.849359 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.849330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" event={"ID":"cd192207-7476-4816-a50d-ec3a230d5d73","Type":"ContainerStarted","Data":"c17fe150d9351acc8ad813c05ad36ec8529f3845d18b21e8e8795bc2cd75febf"}
Jul 24 22:46:26.850462 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.850436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b" event={"ID":"a1f3294d-e87d-4096-a9a9-0e943bd9b98a","Type":"ContainerStarted","Data":"232b905769016111947f6736f2cda72aa42b5ad63669e2cf42dd0bca70198796"}
Jul 24 22:46:26.851497 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.851476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6lphx" event={"ID":"06426582-6029-4f0b-a88f-eea7b0a829d6","Type":"ContainerStarted","Data":"d54df8fbc5f803e472158b82efd596f58dd95e7a38f26396a602de7e03647989"}
Jul 24 22:46:26.852548 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.852508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-85c4k" event={"ID":"31ca8850-9999-4ca9-be05-4dc4f733d845","Type":"ContainerStarted","Data":"bbc9f6f03fe4ea21dcaef3f4c5ea12166dc769e0999e447a7dfa48407babf486"}
Jul 24 22:46:26.853606 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.853580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8qv9m" event={"ID":"1746104c-a257-4d75-ba28-02305cbb01ba","Type":"ContainerStarted","Data":"e3b2529d3c34dbbc3eb50604f2be7ecddebd88aa44185f8db276fd4ef0676b0b"}
Jul 24 22:46:26.858133 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.858105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"70ef009a36b92bc781176538dc2d95ad376cd47b791fc68b43dac103992fe6f1"}
Jul 24 22:46:26.859166 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.859148    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2gwg8" event={"ID":"f8634fe9-3c65-480b-ab1c-ab306e77b9f6","Type":"ContainerStarted","Data":"06ff3daf8718b37dce6293b609c3c560f7e39d7363acdb40607a3898b488cd9d"}
Jul 24 22:46:26.860077 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.860055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cfv6k" event={"ID":"7e316233-25e3-4089-9d90-006bf80dbc8f","Type":"ContainerStarted","Data":"fad6a3229d4a4b55f3140a7497799a2a646d5d47bcbce374334c05903f7326ec"}
Jul 24 22:46:26.860742 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:26.860697    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-221.ec2.internal" podStartSLOduration=2.8606852590000003 podStartE2EDuration="2.860685259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:26.860453509 +0000 UTC m=+3.493732449" watchObservedRunningTime="2026-07-24 22:46:26.860685259 +0000 UTC m=+3.493964199"
Jul 24 22:46:27.330592 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.330549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:27.330793 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.330771    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:27.330878 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.330843    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs podName:35f71564-7bff-4960-a63b-584726e054eb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:29.330823062 +0000 UTC m=+5.964101995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs") pod "network-metrics-daemon-nspfm" (UID: "35f71564-7bff-4960-a63b-584726e054eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:27.431626 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.431582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:27.431813 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.431777    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:27.431878 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.431843    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret podName:303ae2ca-2b38-4094-9507-21820dc52eeb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:29.431824009 +0000 UTC m=+6.065102946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret") pod "global-pull-secret-syncer-rv5zf" (UID: "303ae2ca-2b38-4094-9507-21820dc52eeb") : object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:27.532753 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.532713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:27.532940 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.532911    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 22:46:27.532940 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.532933    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 22:46:27.533044 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.532946    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tts6v for pod openshift-network-diagnostics/network-check-target-48cpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:27.533044 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.533005    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v podName:7fb93a54-2070-48f9-9810-e3f4acb7250c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:29.532986604 +0000 UTC m=+6.166265539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tts6v" (UniqueName: "kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v") pod "network-check-target-48cpz" (UID: "7fb93a54-2070-48f9-9810-e3f4acb7250c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:27.745505 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.745436    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:41:25 +0000 UTC" deadline="2028-01-02 23:31:09.581104201 +0000 UTC"
Jul 24 22:46:27.745505 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.745502    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12648h44m41.835606143s"
Jul 24 22:46:27.837972 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.837670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:27.837972 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.837819    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:27.838280 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.838236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:27.838605 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:27.838337    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:27.886646 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.885575    2563 generic.go:356] "Generic (PLEG): container finished" podID="29ce5794fceb53a7c97a5acddf19c77a" containerID="705709780cd05b6b385614229d525c7aa4f764d630b13efca4ea7cec5e2083e0" exitCode=0
Jul 24 22:46:27.886646 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:27.886580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal" event={"ID":"29ce5794fceb53a7c97a5acddf19c77a","Type":"ContainerDied","Data":"705709780cd05b6b385614229d525c7aa4f764d630b13efca4ea7cec5e2083e0"}
Jul 24 22:46:28.836639 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:28.836172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:28.836639 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:28.836280    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:28.902692 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:28.902582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal" event={"ID":"29ce5794fceb53a7c97a5acddf19c77a","Type":"ContainerStarted","Data":"5bc6b016266b8c1caabe6c4ead6f18e1d65635215f75b114372003187e5b88c9"}
Jul 24 22:46:28.918406 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:28.918342    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-221.ec2.internal" podStartSLOduration=4.918317533 podStartE2EDuration="4.918317533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:28.917933653 +0000 UTC m=+5.551212595" watchObservedRunningTime="2026-07-24 22:46:28.918317533 +0000 UTC m=+5.551596474"
Jul 24 22:46:29.350547 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:29.349716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:29.350547 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.349949    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:29.350547 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.350014    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs podName:35f71564-7bff-4960-a63b-584726e054eb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:33.349995413 +0000 UTC m=+9.983274346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs") pod "network-metrics-daemon-nspfm" (UID: "35f71564-7bff-4960-a63b-584726e054eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:29.450931 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:29.450305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:29.450931 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.450504    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:29.450931 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.450586    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret podName:303ae2ca-2b38-4094-9507-21820dc52eeb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:33.450565629 +0000 UTC m=+10.083844563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret") pod "global-pull-secret-syncer-rv5zf" (UID: "303ae2ca-2b38-4094-9507-21820dc52eeb") : object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:29.552057 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:29.551397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:29.552057 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.551655    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 22:46:29.552057 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.551676    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 22:46:29.552057 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.551689    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tts6v for pod openshift-network-diagnostics/network-check-target-48cpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:29.552057 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.551743    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v podName:7fb93a54-2070-48f9-9810-e3f4acb7250c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:33.551728975 +0000 UTC m=+10.185007905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tts6v" (UniqueName: "kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v") pod "network-check-target-48cpz" (UID: "7fb93a54-2070-48f9-9810-e3f4acb7250c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:29.838200 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:29.838111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:29.838657 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.838250    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:29.838657 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:29.838572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:29.838657 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:29.838652    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:30.836152 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:30.836097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:30.836331 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:30.836250    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:31.836197 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:31.836161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:31.836671 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:31.836164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:31.836671 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:31.836309    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:31.836671 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:31.836394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:32.836064 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:32.836026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:32.836267 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:32.836171    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:33.388225 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:33.388128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:33.388409 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.388304    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:33.388409 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.388384    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs podName:35f71564-7bff-4960-a63b-584726e054eb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:41.388362015 +0000 UTC m=+18.021640947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs") pod "network-metrics-daemon-nspfm" (UID: "35f71564-7bff-4960-a63b-584726e054eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:33.489413 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:33.488807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:33.489413 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.488993    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:33.489413 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.489069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret podName:303ae2ca-2b38-4094-9507-21820dc52eeb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:41.489050462 +0000 UTC m=+18.122329385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret") pod "global-pull-secret-syncer-rv5zf" (UID: "303ae2ca-2b38-4094-9507-21820dc52eeb") : object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:33.589496 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:33.589450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:33.589762 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.589685    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 22:46:33.589762 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.589712    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 22:46:33.589762 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.589725    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tts6v for pod openshift-network-diagnostics/network-check-target-48cpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:33.590077 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.589796    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v podName:7fb93a54-2070-48f9-9810-e3f4acb7250c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:41.589776302 +0000 UTC m=+18.223055221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tts6v" (UniqueName: "kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v") pod "network-check-target-48cpz" (UID: "7fb93a54-2070-48f9-9810-e3f4acb7250c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:33.836931 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:33.836847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:33.837369 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.836983    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:33.837369 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:33.837031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:33.837369 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:33.837133    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:34.836681 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:34.836143    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:34.836681 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:34.836300    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:35.836321 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:35.836284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:35.836857 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:35.836284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:35.836857 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:35.836419    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:35.836857 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:35.836538    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:36.835569 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:36.835515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:36.835742 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:36.835648    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:37.835803 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:37.835766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:37.836274 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:37.835932    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:37.836274 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:37.835943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:37.836274 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:37.836039    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:38.835190 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:38.835147    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:38.835387 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:38.835294    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:39.835816 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:39.835758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:39.836273 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:39.835775    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:39.836273 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:39.835884    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:39.836273 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:39.835965    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:40.835942 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:40.835901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:40.836452 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:40.836043    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:41.451116 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:41.451071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:41.451315 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.451213    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:41.451315 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.451287    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs podName:35f71564-7bff-4960-a63b-584726e054eb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:57.451267512 +0000 UTC m=+34.084546450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs") pod "network-metrics-daemon-nspfm" (UID: "35f71564-7bff-4960-a63b-584726e054eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:41.552001 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:41.551964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:41.552204 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.552148    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:41.552264 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.552218    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret podName:303ae2ca-2b38-4094-9507-21820dc52eeb nodeName:}" failed. No retries permitted until 2026-07-24 22:46:57.552201553 +0000 UTC m=+34.185480475 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret") pod "global-pull-secret-syncer-rv5zf" (UID: "303ae2ca-2b38-4094-9507-21820dc52eeb") : object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:41.653183 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:41.653144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:41.653350 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.653319    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 22:46:41.653350 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.653341    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 22:46:41.653350 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.653350    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tts6v for pod openshift-network-diagnostics/network-check-target-48cpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:41.653442 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.653404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v podName:7fb93a54-2070-48f9-9810-e3f4acb7250c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:57.653389944 +0000 UTC m=+34.286668861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tts6v" (UniqueName: "kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v") pod "network-check-target-48cpz" (UID: "7fb93a54-2070-48f9-9810-e3f4acb7250c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:41.835294 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:41.835210    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:41.835294 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:41.835242    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:41.835542 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.835378    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:41.835542 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:41.835511    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:42.836061 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:42.836023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:42.836553 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:42.836155    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:43.836679 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:43.836617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:43.837134 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:43.836739    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:43.837134 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:43.836801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:43.837828 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:43.837773    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:44.835804 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:44.835768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:44.836014 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:44.835886    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:45.836023 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:45.835253    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:45.836023 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:45.835387    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:45.836023 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:45.835258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:45.836023 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:45.835936    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:45.939214 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:45.939029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6lphx" event={"ID":"06426582-6029-4f0b-a88f-eea7b0a829d6","Type":"ContainerStarted","Data":"b3d9e6941925a425e25a9327997cc6fd80beaaa128d03f91d4bc77e650f6ec7e"}
Jul 24 22:46:45.941263 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:45.941227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-85c4k" event={"ID":"31ca8850-9999-4ca9-be05-4dc4f733d845","Type":"ContainerStarted","Data":"a7c6a51119df7af3f11155739769157d5bf6f4f790458e19f3d02f16f61455c0"}
Jul 24 22:46:45.953088 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:45.953018    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6lphx" podStartSLOduration=3.920727716 podStartE2EDuration="22.9530038s" totalImagesPullingTime="19.032276084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:45.952833286 +0000 UTC m=+22.586112227" watchObservedRunningTime="2026-07-24 22:46:45.9530038 +0000 UTC m=+22.586282739"
Jul 24 22:46:45.966654 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:45.966299    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-85c4k" podStartSLOduration=7.98766177 podStartE2EDuration="22.966278922s" totalImagesPullingTime="14.978617152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:45.966077565 +0000 UTC m=+22.599356508" watchObservedRunningTime="2026-07-24 22:46:45.966278922 +0000 UTC m=+22.599557864"
Jul 24 22:46:46.836077 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.835912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:46.836770 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:46.836155    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:46.945074 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.944995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8qv9m" event={"ID":"1746104c-a257-4d75-ba28-02305cbb01ba","Type":"ContainerStarted","Data":"a1fa2b217fa236832591584db6907dca83c21175dad8ee5cf139dd90de3c44b1"}
Jul 24 22:46:46.947923 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.947898    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 22:46:46.948334 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.948313    2563 generic.go:356] "Generic (PLEG): container finished" podID="4d27a0bc-41c6-480e-a4e9-9c0386da97f0" containerID="796031347c2fa5ad355d04506f5ac03d570c28907d8aad579e65e1cba8ccec11" exitCode=1
Jul 24 22:46:46.948422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.948361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"71b6c0deb00329bf13fc5592df83f0ab1636427df212b58dd066af65ba373bb9"}
Jul 24 22:46:46.948480 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.948461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"c4470a0e1fec714b82fc2aba8a7bd04552168867073abb30341aac85a5b187e7"}
Jul 24 22:46:46.948554 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.948539    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"edd6e6adcd678b15549244d2251c29e2714d52317f427f18b6a293a5989a89cd"}
Jul 24 22:46:46.948624 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.948599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"915bbfcbd33d62ffa735c9c02464eae5aaaf801c004f07d995c55981b1ea7654"}
Jul 24 22:46:46.948728 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.948663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerDied","Data":"796031347c2fa5ad355d04506f5ac03d570c28907d8aad579e65e1cba8ccec11"}
Jul 24 22:46:46.948779 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.948725    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"bb92bb080d457da4b759dcac8aedcc40fafb6cefe8bbfe00d25fee1771887622"}
Jul 24 22:46:46.949851 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.949827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2gwg8" event={"ID":"f8634fe9-3c65-480b-ab1c-ab306e77b9f6","Type":"ContainerStarted","Data":"d38d6d05be2c466107346ceb8cc8073b9a9a7deaeef6353054eb87f53055d37a"}
Jul 24 22:46:46.951287 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.951265    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nrlfg" event={"ID":"9a332a24-918a-4476-b1d4-a013ede37685","Type":"ContainerStarted","Data":"7518d41f48216697e2d66d365f2834d069b44d0f292c6e63ce74a32ae3d7cbfc"}
Jul 24 22:46:46.952799 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.952775    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd192207-7476-4816-a50d-ec3a230d5d73" containerID="4c1679249d07ea8c64c18b0cd25afb3f655f44061d0d2732619e751fc30342f3" exitCode=0
Jul 24 22:46:46.952867 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.952847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" event={"ID":"cd192207-7476-4816-a50d-ec3a230d5d73","Type":"ContainerDied","Data":"4c1679249d07ea8c64c18b0cd25afb3f655f44061d0d2732619e751fc30342f3"}
Jul 24 22:46:46.954242 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.954222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b" event={"ID":"a1f3294d-e87d-4096-a9a9-0e943bd9b98a","Type":"ContainerStarted","Data":"5ee90bdbc2eac3e1752541fd8995f905cb475eb1165c309b717414716fd68698"}
Jul 24 22:46:46.959421 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:46.959379    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8qv9m" podStartSLOduration=4.90582299 podStartE2EDuration="23.959366741s" totalImagesPullingTime="19.053543751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:46.959073974 +0000 UTC m=+23.592352915" watchObservedRunningTime="2026-07-24 22:46:46.959366741 +0000 UTC m=+23.592645680"
Jul 24 22:46:47.023111 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.023063    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nrlfg" podStartSLOduration=4.963773618 podStartE2EDuration="24.02304576s" totalImagesPullingTime="19.059272142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:46.991093387 +0000 UTC m=+23.624372328" watchObservedRunningTime="2026-07-24 22:46:47.02304576 +0000 UTC m=+23.656324741"
Jul 24 22:46:47.041584 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.041510    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2gwg8" podStartSLOduration=4.943995203 podStartE2EDuration="24.041494131s" totalImagesPullingTime="19.097498928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:47.041260557 +0000 UTC m=+23.674539497" watchObservedRunningTime="2026-07-24 22:46:47.041494131 +0000 UTC m=+23.674773071"
Jul 24 22:46:47.067768 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.067731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:47.088670 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.088640    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 22:46:47.161197 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.161163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:47.216171 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.216136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:47.773592 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.773411    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T22:46:47.088666219Z","UUID":"36003e26-b24d-4518-aea8-94079ad8a3db","Handler":null,"Name":"","Endpoint":""}
Jul 24 22:46:47.775104 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.775075    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 22:46:47.775254 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.775126    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 22:46:47.836078 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.836049    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:47.836611 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.836090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:47.836611 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:47.836268    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:47.836761 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:47.836735    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:47.959371 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.958932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cfv6k" event={"ID":"7e316233-25e3-4089-9d90-006bf80dbc8f","Type":"ContainerStarted","Data":"95d04ecbb9c55c931894d17586f049c84ea55a48bbfaccfa72f40d01de06e182"}
Jul 24 22:46:47.961539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.961493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b" event={"ID":"a1f3294d-e87d-4096-a9a9-0e943bd9b98a","Type":"ContainerStarted","Data":"7ad3dc6a3d8a86d6ed66308e31916edee24070a1469eda426286c64c709b13d5"}
Jul 24 22:46:47.974780 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:47.974721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cfv6k" podStartSLOduration=5.912603776 podStartE2EDuration="24.974698568s" totalImagesPullingTime="19.062094792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:47.974399222 +0000 UTC m=+24.607678163" watchObservedRunningTime="2026-07-24 22:46:47.974698568 +0000 UTC m=+24.607977579"
Jul 24 22:46:48.092493 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:48.092456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:48.149009 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:48.148963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:48.835477 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:48.835395    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:48.835666 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:48.835573    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:48.965751 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:48.965714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b" event={"ID":"a1f3294d-e87d-4096-a9a9-0e943bd9b98a","Type":"ContainerStarted","Data":"10d9a1e16a77979fe2ec8abaee667a93c50412348bcd86f8a702c13fa99c59f0"}
Jul 24 22:46:48.988746 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:48.988684    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f446b" podStartSLOduration=4.689525036 podStartE2EDuration="25.988661909s" totalImagesPullingTime="21.299136873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:48.988616568 +0000 UTC m=+25.621895508" watchObservedRunningTime="2026-07-24 22:46:48.988661909 +0000 UTC m=+25.621940850"
Jul 24 22:46:49.378022 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:49.377983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:49.378691 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:49.378668    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:49.839387 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:49.839303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:49.839577 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:49.839427    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:49.839577 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:49.839487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:49.839710 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:49.839649    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:49.972793 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:49.972570    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 22:46:49.973250 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:49.973181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"051c9a994d7cecd501aa8d7623ec2b947be411e94dd000d5677fe0fb7302fcb9"}
Jul 24 22:46:49.973654 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:49.973623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:49.973979 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:49.973955    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-85c4k"
Jul 24 22:46:50.038382 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:50.038348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:50.094514 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:50.094481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:50.151938 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:50.151906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:50.836078 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:50.836039    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:50.836236 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:50.836181    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:51.408423 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.408345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:51.508721 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.508532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:51.835353 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.835315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:51.835549 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.835317    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:51.835549 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:51.835431    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:51.835549 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:51.835498    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:51.979532 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.979488    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd192207-7476-4816-a50d-ec3a230d5d73" containerID="065a48c53a9f7f2c4febcaf3ad258dd50201e39686eb448eb1019908b1988c60" exitCode=0
Jul 24 22:46:51.979723 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.979582    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" event={"ID":"cd192207-7476-4816-a50d-ec3a230d5d73","Type":"ContainerDied","Data":"065a48c53a9f7f2c4febcaf3ad258dd50201e39686eb448eb1019908b1988c60"}
Jul 24 22:46:51.982988 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.982970    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 22:46:51.983361 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.983338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"9ef5c412cb385ca99a7ca2301940be4d468b2ea79e9f469891313bcdd9d1a93a"}
Jul 24 22:46:51.983655 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.983633    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:51.983767 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.983711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:51.983904 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.983888    2563 scope.go:117] "RemoveContainer" containerID="796031347c2fa5ad355d04506f5ac03d570c28907d8aad579e65e1cba8ccec11"
Jul 24 22:46:51.999926 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:51.999898    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:52.390505 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:52.390475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:52.491911 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:52.491877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:52.591970 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:52.591929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:52.835790 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:52.835710    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:52.835974 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:52.835873    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:52.991123 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:52.991096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 22:46:52.991485 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:52.991455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" event={"ID":"4d27a0bc-41c6-480e-a4e9-9c0386da97f0","Type":"ContainerStarted","Data":"26b13cd932817d53a0158a125e5069ad846632a7b444ac24b419e763e131430d"}
Jul 24 22:46:52.991782 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:52.991762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:52.994728 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:52.994699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:53.007913 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.007888    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:46:53.024039 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.023985    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6" podStartSLOduration=10.553540189 podStartE2EDuration="30.023969449s" totalImagesPullingTime="19.47042926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:53.023345026 +0000 UTC m=+29.656623989" watchObservedRunningTime="2026-07-24 22:46:53.023969449 +0000 UTC m=+29.657248389"
Jul 24 22:46:53.059538 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.059473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:53.505461 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.505375    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rv5zf"]
Jul 24 22:46:53.505985 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.505505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:53.505985 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:53.505653    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:53.507642 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.507604    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nspfm"]
Jul 24 22:46:53.507760 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.507744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:53.507864 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:53.507842    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:53.508194 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.508168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-48cpz"]
Jul 24 22:46:53.508277 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.508266    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:53.508383 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:53.508358    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:53.995946 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.995911    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd192207-7476-4816-a50d-ec3a230d5d73" containerID="d6d86a5fa80911696be9a2811abfec7f88847d9baa9352979b51d0867796482b" exitCode=0
Jul 24 22:46:53.996095 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:53.995991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" event={"ID":"cd192207-7476-4816-a50d-ec3a230d5d73","Type":"ContainerDied","Data":"d6d86a5fa80911696be9a2811abfec7f88847d9baa9352979b51d0867796482b"}
Jul 24 22:46:54.779322 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:54.779292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:54.879044 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:54.879010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:55.836260 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:55.836010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:55.836760 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:55.836041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:55.836760 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:55.836377    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:55.836760 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:55.836041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:55.836760 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:55.836435    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:55.836760 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:55.836507    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:56.002469 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.002436    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd192207-7476-4816-a50d-ec3a230d5d73" containerID="a4fa1ef3b0cadef288a426990e0b01aa595658042b99d7de55f4c263c40e6081" exitCode=0
Jul 24 22:46:56.002647 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.002515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" event={"ID":"cd192207-7476-4816-a50d-ec3a230d5d73","Type":"ContainerDied","Data":"a4fa1ef3b0cadef288a426990e0b01aa595658042b99d7de55f4c263c40e6081"}
Jul 24 22:46:56.091539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.091483    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:56.191264 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.191233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:56.291024 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.290985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:56.423506 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.423420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:56.477785 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.477750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:56.790062 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.789982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:56.862216 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.862179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:56.914383 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:56.914346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:57.139784 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:57.139747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:57.235351 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:57.235319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:57.475084 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:57.474989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:57.475240 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.475165    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:57.475304 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.475247    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs podName:35f71564-7bff-4960-a63b-584726e054eb nodeName:}" failed. No retries permitted until 2026-07-24 22:47:29.475226182 +0000 UTC m=+66.108505114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs") pod "network-metrics-daemon-nspfm" (UID: "35f71564-7bff-4960-a63b-584726e054eb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 22:46:57.575845 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:57.575810    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:57.576016 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.575999    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:57.576095 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.576081    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret podName:303ae2ca-2b38-4094-9507-21820dc52eeb nodeName:}" failed. No retries permitted until 2026-07-24 22:47:29.576060122 +0000 UTC m=+66.209339050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret") pod "global-pull-secret-syncer-rv5zf" (UID: "303ae2ca-2b38-4094-9507-21820dc52eeb") : object "kube-system"/"original-pull-secret" not registered
Jul 24 22:46:57.676738 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:57.676697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:57.676907 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.676880    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 22:46:57.676907 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.676905    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 22:46:57.677021 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.676918    2563 projected.go:196] Error preparing data for projected volume kube-api-access-tts6v for pod openshift-network-diagnostics/network-check-target-48cpz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:57.677021 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.676988    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v podName:7fb93a54-2070-48f9-9810-e3f4acb7250c nodeName:}" failed. No retries permitted until 2026-07-24 22:47:29.676966609 +0000 UTC m=+66.310245541 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tts6v" (UniqueName: "kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v") pod "network-check-target-48cpz" (UID: "7fb93a54-2070-48f9-9810-e3f4acb7250c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 22:46:57.835468 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:57.835380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:57.835639 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:57.835382    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:57.835639 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.835510    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nspfm" podUID="35f71564-7bff-4960-a63b-584726e054eb"
Jul 24 22:46:57.835639 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:57.835386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:57.835639 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.835622    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rv5zf" podUID="303ae2ca-2b38-4094-9507-21820dc52eeb"
Jul 24 22:46:57.835800 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:57.835643    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-48cpz" podUID="7fb93a54-2070-48f9-9810-e3f4acb7250c"
Jul 24 22:46:58.157819 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.157794    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6lphx_06426582-6029-4f0b-a88f-eea7b0a829d6/dns-node-resolver/0.log"
Jul 24 22:46:58.669911 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.669880    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-221.ec2.internal" event="NodeReady"
Jul 24 22:46:58.670194 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.670042    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 22:46:58.712177 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.712144    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-blcb4"]
Jul 24 22:46:58.714941 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.714609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.715721 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.715698    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r5g4j"]
Jul 24 22:46:58.717122 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.717097    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l2hpm\""
Jul 24 22:46:58.717247 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.717097    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 22:46:58.717479 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.717404    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 22:46:58.717479 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.717473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r5g4j"
Jul 24 22:46:58.720170 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.720146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69gfn\""
Jul 24 22:46:58.720295 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.720182    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 22:46:58.720295 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.720227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 22:46:58.721171 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.721139    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 22:46:58.726261 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.726111    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-blcb4"]
Jul 24 22:46:58.728797 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.728769    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r5g4j"]
Jul 24 22:46:58.731184 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.731162    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rtc6q"]
Jul 24 22:46:58.733837 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.733814    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.736544 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.736497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 24 22:46:58.736652 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.736604    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 24 22:46:58.736652 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.736614    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 24 22:46:58.736865 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.736836    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 24 22:46:58.737391 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.737372    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rpl77\""
Jul 24 22:46:58.739500 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.739472    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jzzs7"]
Jul 24 22:46:58.741425 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.741403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.744094 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.744066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rtc6q"]
Jul 24 22:46:58.744279 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.744224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 22:46:58.744279 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.744262    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlc68\""
Jul 24 22:46:58.744710 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.744691    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 22:46:58.744801 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.744758    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 22:46:58.744801 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.744789    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 22:46:58.752460 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.752431    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jzzs7"]
Jul 24 22:46:58.774989 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.774951    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:58.785059 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.785028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfz7\" (UniqueName: \"kubernetes.io/projected/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-kube-api-access-zwfz7\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.785253 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.785111    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-config-volume\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.785253 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.785148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbe5e964-56ec-4897-8b14-7d09a515919e-cert\") pod \"ingress-canary-r5g4j\" (UID: \"fbe5e964-56ec-4897-8b14-7d09a515919e\") " pod="openshift-ingress-canary/ingress-canary-r5g4j"
Jul 24 22:46:58.785253 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.785184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.785253 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.785208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-tmp-dir\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.785253 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.785224    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln59s\" (UniqueName: \"kubernetes.io/projected/fbe5e964-56ec-4897-8b14-7d09a515919e-kube-api-access-ln59s\") pod \"ingress-canary-r5g4j\" (UID: \"fbe5e964-56ec-4897-8b14-7d09a515919e\") " pod="openshift-ingress-canary/ingress-canary-r5g4j"
Jul 24 22:46:58.832092 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.832051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:58.886251 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.886251 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-tmp-dir\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.886494 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln59s\" (UniqueName: \"kubernetes.io/projected/fbe5e964-56ec-4897-8b14-7d09a515919e-kube-api-access-ln59s\") pod \"ingress-canary-r5g4j\" (UID: \"fbe5e964-56ec-4897-8b14-7d09a515919e\") " pod="openshift-ingress-canary/ingress-canary-r5g4j"
Jul 24 22:46:58.886494 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-data-volume\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.886494 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m79bz\" (UniqueName: \"kubernetes.io/projected/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-api-access-m79bz\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.886494 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:58.886342    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 22:46:58.886494 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfz7\" (UniqueName: \"kubernetes.io/projected/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-kube-api-access-zwfz7\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.886494 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.886494 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:58.886425    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls podName:2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca nodeName:}" failed. No retries permitted until 2026-07-24 22:46:59.386405769 +0000 UTC m=+36.019684689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls") pod "dns-default-blcb4" (UID: "2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca") : secret "dns-default-metrics-tls" not found
Jul 24 22:46:58.886850 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-tmp-dir\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.886850 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58cdbb0f-1467-4bac-8335-d2216c8d093f-signing-key\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.886850 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58cdbb0f-1467-4bac-8335-d2216c8d093f-signing-cabundle\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.886850 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87ws\" (UniqueName: \"kubernetes.io/projected/58cdbb0f-1467-4bac-8335-d2216c8d093f-kube-api-access-f87ws\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.887042 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-config-volume\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.887042 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbe5e964-56ec-4897-8b14-7d09a515919e-cert\") pod \"ingress-canary-r5g4j\" (UID: \"fbe5e964-56ec-4897-8b14-7d09a515919e\") " pod="openshift-ingress-canary/ingress-canary-r5g4j"
Jul 24 22:46:58.887042 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.886980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-crio-socket\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.887042 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.887007    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.887383 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.887363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-config-volume\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.891776 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.891748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbe5e964-56ec-4897-8b14-7d09a515919e-cert\") pod \"ingress-canary-r5g4j\" (UID: \"fbe5e964-56ec-4897-8b14-7d09a515919e\") " pod="openshift-ingress-canary/ingress-canary-r5g4j"
Jul 24 22:46:58.897313 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.897285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfz7\" (UniqueName: \"kubernetes.io/projected/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-kube-api-access-zwfz7\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:58.897515 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.897491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln59s\" (UniqueName: \"kubernetes.io/projected/fbe5e964-56ec-4897-8b14-7d09a515919e-kube-api-access-ln59s\") pod \"ingress-canary-r5g4j\" (UID: \"fbe5e964-56ec-4897-8b14-7d09a515919e\") " pod="openshift-ingress-canary/ingress-canary-r5g4j"
Jul 24 22:46:58.987615 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.987502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58cdbb0f-1467-4bac-8335-d2216c8d093f-signing-cabundle\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.987615 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.987586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87ws\" (UniqueName: \"kubernetes.io/projected/58cdbb0f-1467-4bac-8335-d2216c8d093f-kube-api-access-f87ws\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.987857 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.987670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-crio-socket\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.987857 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.987702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.987857 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.987747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-data-volume\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.987857 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.987776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m79bz\" (UniqueName: \"kubernetes.io/projected/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-api-access-m79bz\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.987986 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:58.987906    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 22:46:58.987986 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.987928    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-crio-socket\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.987986 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:58.987972    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls podName:310b20a4-9c4c-44ec-bb15-f2c86b76e64a nodeName:}" failed. No retries permitted until 2026-07-24 22:46:59.487956491 +0000 UTC m=+36.121235413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jzzs7" (UID: "310b20a4-9c4c-44ec-bb15-f2c86b76e64a") : secret "insights-runtime-extractor-tls" not found
Jul 24 22:46:58.988128 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.987994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.988128 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.988031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58cdbb0f-1467-4bac-8335-d2216c8d093f-signing-key\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.988128 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:58.988118    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 22:46:58.988259 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:58.988178    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm podName:310b20a4-9c4c-44ec-bb15-f2c86b76e64a nodeName:}" failed. No retries permitted until 2026-07-24 22:46:59.488163186 +0000 UTC m=+36.121442105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jzzs7" (UID: "310b20a4-9c4c-44ec-bb15-f2c86b76e64a") : configmap "kube-rbac-proxy" not found
Jul 24 22:46:58.988259 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.988194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-data-volume\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.988328 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.988314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/58cdbb0f-1467-4bac-8335-d2216c8d093f-signing-cabundle\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.990748 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.990725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/58cdbb0f-1467-4bac-8335-d2216c8d093f-signing-key\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:58.996997 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.996959    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m79bz\" (UniqueName: \"kubernetes.io/projected/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-api-access-m79bz\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:58.997185 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:58.997160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87ws\" (UniqueName: \"kubernetes.io/projected/58cdbb0f-1467-4bac-8335-d2216c8d093f-kube-api-access-f87ws\") pod \"service-ca-658d7b7c78-rtc6q\" (UID: \"58cdbb0f-1467-4bac-8335-d2216c8d093f\") " pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:59.036080 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.036041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r5g4j"
Jul 24 22:46:59.046680 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.046645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q"
Jul 24 22:46:59.158325 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.158298    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8qv9m_1746104c-a257-4d75-ba28-02305cbb01ba/node-ca/0.log"
Jul 24 22:46:59.391825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.391788    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:46:59.392004 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:59.391948    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 22:46:59.392064 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:59.392011    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls podName:2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca nodeName:}" failed. No retries permitted until 2026-07-24 22:47:00.391995654 +0000 UTC m=+37.025274596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls") pod "dns-default-blcb4" (UID: "2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca") : secret "dns-default-metrics-tls" not found
Jul 24 22:46:59.493193 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.493152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:59.493386 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.493241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:46:59.493386 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:59.493329    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 22:46:59.493386 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:59.493349    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 22:46:59.493545 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:59.493406    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls podName:310b20a4-9c4c-44ec-bb15-f2c86b76e64a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:00.493388896 +0000 UTC m=+37.126667814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jzzs7" (UID: "310b20a4-9c4c-44ec-bb15-f2c86b76e64a") : secret "insights-runtime-extractor-tls" not found
Jul 24 22:46:59.493545 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:46:59.493423    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm podName:310b20a4-9c4c-44ec-bb15-f2c86b76e64a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:00.493415788 +0000 UTC m=+37.126694706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jzzs7" (UID: "310b20a4-9c4c-44ec-bb15-f2c86b76e64a") : configmap "kube-rbac-proxy" not found
Jul 24 22:46:59.560441 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.560407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:46:59.835922 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.835834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:46:59.836086 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.835836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:46:59.836086 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.835836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:46:59.838919 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.838868    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 22:46:59.840352 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.840098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 22:46:59.840352 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.840098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmr4r\""
Jul 24 22:46:59.840352 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.840145    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b4wx4\""
Jul 24 22:46:59.840352 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.840164    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 22:46:59.840352 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:46:59.840150    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 22:47:00.318739 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:00.318467    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r5g4j"]
Jul 24 22:47:00.321879 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:00.321850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-rtc6q"]
Jul 24 22:47:00.326300 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:00.326273    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe5e964_56ec_4897_8b14_7d09a515919e.slice/crio-dbc65e3939e46e01b94e64e14a585ee5ab0936b7e5b3987fee4e344d80f9a28e WatchSource:0}: Error finding container dbc65e3939e46e01b94e64e14a585ee5ab0936b7e5b3987fee4e344d80f9a28e: Status 404 returned error can't find the container with id dbc65e3939e46e01b94e64e14a585ee5ab0936b7e5b3987fee4e344d80f9a28e
Jul 24 22:47:00.327276 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:00.327252    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58cdbb0f_1467_4bac_8335_d2216c8d093f.slice/crio-f99370ae0c17b050f0f68693f746722327f0ea31613fcde07a6508f5c94078e1 WatchSource:0}: Error finding container f99370ae0c17b050f0f68693f746722327f0ea31613fcde07a6508f5c94078e1: Status 404 returned error can't find the container with id f99370ae0c17b050f0f68693f746722327f0ea31613fcde07a6508f5c94078e1
Jul 24 22:47:00.400730 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:00.400701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:47:00.400862 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:00.400848    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 22:47:00.400935 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:00.400922    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls podName:2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.40090205 +0000 UTC m=+39.034180988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls") pod "dns-default-blcb4" (UID: "2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca") : secret "dns-default-metrics-tls" not found
Jul 24 22:47:00.501696 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:00.501658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:00.501872 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:00.501733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:00.501872 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:00.501806    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 24 22:47:00.501872 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:00.501833    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 22:47:00.501970 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:00.501892    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm podName:310b20a4-9c4c-44ec-bb15-f2c86b76e64a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.501868485 +0000 UTC m=+39.135147407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jzzs7" (UID: "310b20a4-9c4c-44ec-bb15-f2c86b76e64a") : configmap "kube-rbac-proxy" not found
Jul 24 22:47:00.501970 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:00.501913    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls podName:310b20a4-9c4c-44ec-bb15-f2c86b76e64a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.501902499 +0000 UTC m=+39.135181418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jzzs7" (UID: "310b20a4-9c4c-44ec-bb15-f2c86b76e64a") : secret "insights-runtime-extractor-tls" not found
Jul 24 22:47:01.017550 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:01.017495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r5g4j" event={"ID":"fbe5e964-56ec-4897-8b14-7d09a515919e","Type":"ContainerStarted","Data":"dbc65e3939e46e01b94e64e14a585ee5ab0936b7e5b3987fee4e344d80f9a28e"}
Jul 24 22:47:01.021100 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:01.021057    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd192207-7476-4816-a50d-ec3a230d5d73" containerID="7a43ba21d93075341143541ef864db309d3619e288b571876cb15c12e2f6ba91" exitCode=0
Jul 24 22:47:01.021266 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:01.021140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" event={"ID":"cd192207-7476-4816-a50d-ec3a230d5d73","Type":"ContainerDied","Data":"7a43ba21d93075341143541ef864db309d3619e288b571876cb15c12e2f6ba91"}
Jul 24 22:47:01.022567 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:01.022511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q" event={"ID":"58cdbb0f-1467-4bac-8335-d2216c8d093f","Type":"ContainerStarted","Data":"f99370ae0c17b050f0f68693f746722327f0ea31613fcde07a6508f5c94078e1"}
Jul 24 22:47:01.987544 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:01.987489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:02.028920 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:02.028882    2563 generic.go:356] "Generic (PLEG): container finished" podID="cd192207-7476-4816-a50d-ec3a230d5d73" containerID="06cec45409c830490697518b2e2354d565579f2b98def77da7afd02d787fbaa6" exitCode=0
Jul 24 22:47:02.029088 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:02.028928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" event={"ID":"cd192207-7476-4816-a50d-ec3a230d5d73","Type":"ContainerDied","Data":"06cec45409c830490697518b2e2354d565579f2b98def77da7afd02d787fbaa6"}
Jul 24 22:47:02.419919 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:02.419874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:47:02.420133 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:02.420014    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 22:47:02.420133 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:02.420086    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls podName:2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca nodeName:}" failed. No retries permitted until 2026-07-24 22:47:06.420068181 +0000 UTC m=+43.053347100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls") pod "dns-default-blcb4" (UID: "2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca") : secret "dns-default-metrics-tls" not found
Jul 24 22:47:02.476612 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:02.476579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:02.521069 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:02.521036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:02.521280 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:02.521116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:02.521280 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:02.521204    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 22:47:02.521394 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:02.521288    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls podName:310b20a4-9c4c-44ec-bb15-f2c86b76e64a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:06.521266946 +0000 UTC m=+43.154545879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jzzs7" (UID: "310b20a4-9c4c-44ec-bb15-f2c86b76e64a") : secret "insights-runtime-extractor-tls" not found
Jul 24 22:47:02.521756 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:02.521726    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:03.118963 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:03.118710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:03.219388 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:03.219353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:04.040670 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.040634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r5g4j" event={"ID":"fbe5e964-56ec-4897-8b14-7d09a515919e","Type":"ContainerStarted","Data":"bd17eecaee844b761ec6db25d9a1febeb958c30a9933da50874c8390b5767787"}
Jul 24 22:47:04.044167 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.044128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" event={"ID":"cd192207-7476-4816-a50d-ec3a230d5d73","Type":"ContainerStarted","Data":"6db6dedb3c3324cea024fc25681f50ee7765bbe6afb5d63d0e040a67037a7bd0"}
Jul 24 22:47:04.045837 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.045807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q" event={"ID":"58cdbb0f-1467-4bac-8335-d2216c8d093f","Type":"ContainerStarted","Data":"aa4271cd1af81572bcad310715ccc6badcbde19d8079b5915edabeb2f7cf56e8"}
Jul 24 22:47:04.058005 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.057946    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r5g4j" podStartSLOduration=3.408495002 podStartE2EDuration="6.057924652s" totalImagesPullingTime="2.64942965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:04.055848773 +0000 UTC m=+40.689127714" watchObservedRunningTime="2026-07-24 22:47:04.057924652 +0000 UTC m=+40.691203595"
Jul 24 22:47:04.078440 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.078390    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5bf8x" podStartSLOduration=11.231369718 podStartE2EDuration="41.07837003s" totalImagesPullingTime="29.847000312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:04.076687309 +0000 UTC m=+40.709966250" watchObservedRunningTime="2026-07-24 22:47:04.07837003 +0000 UTC m=+40.711648972"
Jul 24 22:47:04.091791 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.091736    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-rtc6q" podStartSLOduration=3.447299183 podStartE2EDuration="6.091715045s" totalImagesPullingTime="2.644415862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:04.09079322 +0000 UTC m=+40.724072160" watchObservedRunningTime="2026-07-24 22:47:04.091715045 +0000 UTC m=+40.724993987"
Jul 24 22:47:04.434892 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.434859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:04.819716 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.819623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:04.920137 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:04.920103    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:05.020038 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:05.019986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:05.082343 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:05.082255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:05.170186 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:05.170145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:05.269657 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:05.269620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:06.271262 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:06.271228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:06.405738 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:06.405708    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:06.454058 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:06.454021    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:47:06.454219 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:06.454139    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 22:47:06.454219 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:06.454194    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls podName:2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca nodeName:}" failed. No retries permitted until 2026-07-24 22:47:14.454179211 +0000 UTC m=+51.087458129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls") pod "dns-default-blcb4" (UID: "2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca") : secret "dns-default-metrics-tls" not found
Jul 24 22:47:06.554758 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:06.554660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:06.554908 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:06.554782    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 24 22:47:06.554908 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:06.554861    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls podName:310b20a4-9c4c-44ec-bb15-f2c86b76e64a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:14.55484394 +0000 UTC m=+51.188122858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jzzs7" (UID: "310b20a4-9c4c-44ec-bb15-f2c86b76e64a") : secret "insights-runtime-extractor-tls" not found
Jul 24 22:47:06.609538 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:06.609495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:06.853199 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:06.853169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:07.014277 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:07.014249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:07.210994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:07.210910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:08.010203 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:08.010160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:08.119952 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:08.119922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:08.301332 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:08.301247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:08.407992 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:08.407954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:08.808636 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:08.808602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:09.008591 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:09.008562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:09.060739 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:09.060653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:09.206851 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:09.206819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:09.765971 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:09.765941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:09.866230 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:09.866196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:10.206981 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:10.206950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:10.407218 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:10.407181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:10.710110 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:10.710067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:10.810597 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:10.810563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:11.006335 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:11.006248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:11.404150 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:11.404116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:12.009598 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:12.009532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:12.407296 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:12.407266    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:12.810004 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:12.809922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:13.007894 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:13.007869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:13.604637 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:13.604604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:14.009196 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.009109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:14.207210 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.207176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:14.518536 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.518483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:47:14.522796 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.522766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca-metrics-tls\") pod \"dns-default-blcb4\" (UID: \"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca\") " pod="openshift-dns/dns-default-blcb4"
Jul 24 22:47:14.608536 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.608497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:14.619673 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.619640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:14.621962 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.621938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/310b20a4-9c4c-44ec-bb15-f2c86b76e64a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzzs7\" (UID: \"310b20a4-9c4c-44ec-bb15-f2c86b76e64a\") " pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:14.627930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.627894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-blcb4"
Jul 24 22:47:14.655152 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.655121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jzzs7"
Jul 24 22:47:14.792396 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.792313    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-blcb4"]
Jul 24 22:47:14.795680 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:14.795646    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6a49d4_1ac6_49d1_b540_bf2ad1d7b8ca.slice/crio-d175666a5932e3e535019f921f2f3510708c8e24c851f4ecfa373d19050c5629 WatchSource:0}: Error finding container d175666a5932e3e535019f921f2f3510708c8e24c851f4ecfa373d19050c5629: Status 404 returned error can't find the container with id d175666a5932e3e535019f921f2f3510708c8e24c851f4ecfa373d19050c5629
Jul 24 22:47:14.819736 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:14.818591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jzzs7"]
Jul 24 22:47:14.823308 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:14.823274    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310b20a4_9c4c_44ec_bb15_f2c86b76e64a.slice/crio-b3162506e06a8f3a06689220984aa5190672d4d80d5f23315ced10e518d6fe2a WatchSource:0}: Error finding container b3162506e06a8f3a06689220984aa5190672d4d80d5f23315ced10e518d6fe2a: Status 404 returned error can't find the container with id b3162506e06a8f3a06689220984aa5190672d4d80d5f23315ced10e518d6fe2a
Jul 24 22:47:15.008640 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:15.008597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:15.075059 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:15.074969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blcb4" event={"ID":"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca","Type":"ContainerStarted","Data":"d175666a5932e3e535019f921f2f3510708c8e24c851f4ecfa373d19050c5629"}
Jul 24 22:47:15.076166 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:15.076137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzzs7" event={"ID":"310b20a4-9c4c-44ec-bb15-f2c86b76e64a","Type":"ContainerStarted","Data":"b3162506e06a8f3a06689220984aa5190672d4d80d5f23315ced10e518d6fe2a"}
Jul 24 22:47:15.211242 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:15.211212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:18.085458 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:18.085422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blcb4" event={"ID":"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca","Type":"ContainerStarted","Data":"c7ebbf2955e77846e6c8713edcfabbd4c3b80c54087a1cc6608a7fbbab485e3c"}
Jul 24 22:47:18.085912 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:18.085485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blcb4" event={"ID":"2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca","Type":"ContainerStarted","Data":"7a0951d50241fbb661d4fcc96f45a81768947a62709e0daf76e216af98279fcd"}
Jul 24 22:47:18.085912 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:18.085540    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-blcb4"
Jul 24 22:47:18.103151 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:18.103076    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-blcb4" podStartSLOduration=17.647799508 podStartE2EDuration="20.103048802s" totalImagesPullingTime="2.455249294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:18.102137112 +0000 UTC m=+54.735416064" watchObservedRunningTime="2026-07-24 22:47:18.103048802 +0000 UTC m=+54.736327743"
Jul 24 22:47:21.097341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:21.097299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzzs7" event={"ID":"310b20a4-9c4c-44ec-bb15-f2c86b76e64a","Type":"ContainerStarted","Data":"b7bc7160becf46ed42bdb60f8dcaa82ad0cd8c4e59a254ce5d2b5aa4a9fd3399"}
Jul 24 22:47:22.048047 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.048017    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"]
Jul 24 22:47:22.055275 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.055233    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.059115 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.059087    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"]
Jul 24 22:47:22.059361 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.059338    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 22:47:22.059456 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.059434    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 22:47:22.059656 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.059640    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 22:47:22.059784 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.059767    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 22:47:22.059849 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.059834    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 22:47:22.060104 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.060081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 22:47:22.060190 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.060121    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 22:47:22.061182 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.061164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"
Jul 24 22:47:22.064027 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.063988    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 22:47:22.064374 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.064345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"]
Jul 24 22:47:22.068722 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.068481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lm8ls\""
Jul 24 22:47:22.073828 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.073767    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"]
Jul 24 22:47:22.084047 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.084013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-hub\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.084229 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.084099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd45z\" (UniqueName: \"kubernetes.io/projected/8870d43a-7366-4272-a81a-2b4e7b5af0c6-kube-api-access-hd45z\") pod \"managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9\" (UID: \"8870d43a-7366-4272-a81a-2b4e7b5af0c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"
Jul 24 22:47:22.084229 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.084151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.084229 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.084181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.084401 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.084245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.084401 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.084272    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9l8l\" (UniqueName: \"kubernetes.io/projected/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-kube-api-access-v9l8l\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.084401 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.084303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-ca\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.084401 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.084327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8870d43a-7366-4272-a81a-2b4e7b5af0c6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9\" (UID: \"8870d43a-7366-4272-a81a-2b4e7b5af0c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"
Jul 24 22:47:22.175963 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.175929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:22.185034 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.185000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd45z\" (UniqueName: \"kubernetes.io/projected/8870d43a-7366-4272-a81a-2b4e7b5af0c6-kube-api-access-hd45z\") pod \"managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9\" (UID: \"8870d43a-7366-4272-a81a-2b4e7b5af0c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"
Jul 24 22:47:22.185229 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.185044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.185229 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.185171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.185341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.185254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.185341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.185282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9l8l\" (UniqueName: \"kubernetes.io/projected/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-kube-api-access-v9l8l\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.185736 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.185713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-ca\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.185825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.185762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8870d43a-7366-4272-a81a-2b4e7b5af0c6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9\" (UID: \"8870d43a-7366-4272-a81a-2b4e7b5af0c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"
Jul 24 22:47:22.185880 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.185834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-hub\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.186398 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.186367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.188737 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.188710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.189320 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.189263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.190456 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.190425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-ca\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.191771 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.191725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8870d43a-7366-4272-a81a-2b4e7b5af0c6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9\" (UID: \"8870d43a-7366-4272-a81a-2b4e7b5af0c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"
Jul 24 22:47:22.192500 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.192477    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-hub\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.195195 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.195151    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9l8l\" (UniqueName: \"kubernetes.io/projected/ad093a13-4d31-4dc6-a29d-b7a5b80e23bf-kube-api-access-v9l8l\") pod \"cluster-proxy-proxy-agent-6f685fb595-65fz9\" (UID: \"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.198061 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.198036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd45z\" (UniqueName: \"kubernetes.io/projected/8870d43a-7366-4272-a81a-2b4e7b5af0c6-kube-api-access-hd45z\") pod \"managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9\" (UID: \"8870d43a-7366-4272-a81a-2b4e7b5af0c6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"
Jul 24 22:47:22.278628 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.278591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:22.362534 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.362462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:22.379574 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.379514    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"
Jul 24 22:47:22.384787 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.384757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"
Jul 24 22:47:22.436256 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.436225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:22.810245 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.810171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:22.910469 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:22.910436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:24.207954 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:24.207890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9"]
Jul 24 22:47:24.215400 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:24.215372    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8870d43a_7366_4272_a81a_2b4e7b5af0c6.slice/crio-4740b22343338e316f53bd57632205938b197e5a0c389dbd692c2d8504a5a6a2 WatchSource:0}: Error finding container 4740b22343338e316f53bd57632205938b197e5a0c389dbd692c2d8504a5a6a2: Status 404 returned error can't find the container with id 4740b22343338e316f53bd57632205938b197e5a0c389dbd692c2d8504a5a6a2
Jul 24 22:47:24.222230 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:24.222189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9"]
Jul 24 22:47:24.224931 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:24.224908    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad093a13_4d31_4dc6_a29d_b7a5b80e23bf.slice/crio-3ca954044c2588451c03055c5ea0ce8c2f16a53a8caaf3c50bdd338d368d0fee WatchSource:0}: Error finding container 3ca954044c2588451c03055c5ea0ce8c2f16a53a8caaf3c50bdd338d368d0fee: Status 404 returned error can't find the container with id 3ca954044c2588451c03055c5ea0ce8c2f16a53a8caaf3c50bdd338d368d0fee
Jul 24 22:47:25.014337 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.014309    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sqlz6"
Jul 24 22:47:25.118987 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.118902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzzs7" event={"ID":"310b20a4-9c4c-44ec-bb15-f2c86b76e64a","Type":"ContainerStarted","Data":"29df1321e7c585960cde9b48d6ede20701687bdd5b1b4d6cad0305910257a77a"}
Jul 24 22:47:25.121002 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.120968    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9" event={"ID":"8870d43a-7366-4272-a81a-2b4e7b5af0c6","Type":"ContainerStarted","Data":"4740b22343338e316f53bd57632205938b197e5a0c389dbd692c2d8504a5a6a2"}
Jul 24 22:47:25.123884 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.123824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9" event={"ID":"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf","Type":"ContainerStarted","Data":"3ca954044c2588451c03055c5ea0ce8c2f16a53a8caaf3c50bdd338d368d0fee"}
Jul 24 22:47:25.135480 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.135423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:25.209512 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.209424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:25.279204 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.279111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:25.473928 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.473831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:25.586753 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.586623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:25.645483 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:25.645449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:26.643537 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:26.643486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:26.700289 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:26.700238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:27.419800 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:27.419765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:27.580880 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:27.580840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:27.691512 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:27.691419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:28.091221 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:28.091062    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-blcb4"
Jul 24 22:47:28.262603 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:28.262502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:28.318682 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:28.318651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:29.548599 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.548557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:47:29.551552 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.551529    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 22:47:29.561401 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.561375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35f71564-7bff-4960-a63b-584726e054eb-metrics-certs\") pod \"network-metrics-daemon-nspfm\" (UID: \"35f71564-7bff-4960-a63b-584726e054eb\") " pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:47:29.649712 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.649669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:47:29.652768 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.652740    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 22:47:29.662489 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.662450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/303ae2ca-2b38-4094-9507-21820dc52eeb-original-pull-secret\") pod \"global-pull-secret-syncer-rv5zf\" (UID: \"303ae2ca-2b38-4094-9507-21820dc52eeb\") " pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:47:29.750079 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.750027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:47:29.753251 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.753220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 22:47:29.763312 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.763286    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 22:47:29.774535 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.774484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tts6v\" (UniqueName: \"kubernetes.io/projected/7fb93a54-2070-48f9-9810-e3f4acb7250c-kube-api-access-tts6v\") pod \"network-check-target-48cpz\" (UID: \"7fb93a54-2070-48f9-9810-e3f4acb7250c\") " pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:47:29.850927 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.850897    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b4wx4\""
Jul 24 22:47:29.857914 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.857878    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmr4r\""
Jul 24 22:47:29.858903 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.858883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:47:29.861000 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.860975    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rv5zf"
Jul 24 22:47:29.865767 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:29.865740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nspfm"
Jul 24 22:47:30.821208 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:30.821168    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nspfm"]
Jul 24 22:47:30.826806 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:30.826778    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f71564_7bff_4960_a63b_584726e054eb.slice/crio-800b8cb38f4b7d785b3b4b233f4c566ab8dee9d0d28ca0fe5a43b1a54563e004 WatchSource:0}: Error finding container 800b8cb38f4b7d785b3b4b233f4c566ab8dee9d0d28ca0fe5a43b1a54563e004: Status 404 returned error can't find the container with id 800b8cb38f4b7d785b3b4b233f4c566ab8dee9d0d28ca0fe5a43b1a54563e004
Jul 24 22:47:31.047473 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.047442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rv5zf"]
Jul 24 22:47:31.050948 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.050918    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-48cpz"]
Jul 24 22:47:31.051111 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:31.051053    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303ae2ca_2b38_4094_9507_21820dc52eeb.slice/crio-b7e9961e6f86a6816840d12dd3ca7623cf2b3761ed0e98aadbba680ab6be4c10 WatchSource:0}: Error finding container b7e9961e6f86a6816840d12dd3ca7623cf2b3761ed0e98aadbba680ab6be4c10: Status 404 returned error can't find the container with id b7e9961e6f86a6816840d12dd3ca7623cf2b3761ed0e98aadbba680ab6be4c10
Jul 24 22:47:31.054438 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:31.054413    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb93a54_2070_48f9_9810_e3f4acb7250c.slice/crio-9780aabc98b73324ed93e8b2c9e308638c60782f831eea717b58504e99346b90 WatchSource:0}: Error finding container 9780aabc98b73324ed93e8b2c9e308638c60782f831eea717b58504e99346b90: Status 404 returned error can't find the container with id 9780aabc98b73324ed93e8b2c9e308638c60782f831eea717b58504e99346b90
Jul 24 22:47:31.146483 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.146443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9" event={"ID":"8870d43a-7366-4272-a81a-2b4e7b5af0c6","Type":"ContainerStarted","Data":"4609d88cc7199180a9b357daf41bb4be39cc3463f1e4a9c11263885d1d7507f3"}
Jul 24 22:47:31.148659 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.148624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9" event={"ID":"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf","Type":"ContainerStarted","Data":"05017eb35f32a7862dcb7731373e539ad82fcf889172c9416eabe61dc3401b11"}
Jul 24 22:47:31.148827 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.148671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9" event={"ID":"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf","Type":"ContainerStarted","Data":"8089320033b1356da1956c715ac81eddaefa4f39b3f567b634d851607f80867f"}
Jul 24 22:47:31.148827 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.148687    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9" event={"ID":"ad093a13-4d31-4dc6-a29d-b7a5b80e23bf","Type":"ContainerStarted","Data":"185d9ba2d9c520b4455b7a8cae0857768f5bd2c6f8f0cfa53559a8017c557599"}
Jul 24 22:47:31.149910 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.149884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rv5zf" event={"ID":"303ae2ca-2b38-4094-9507-21820dc52eeb","Type":"ContainerStarted","Data":"b7e9961e6f86a6816840d12dd3ca7623cf2b3761ed0e98aadbba680ab6be4c10"}
Jul 24 22:47:31.151050 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.151027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nspfm" event={"ID":"35f71564-7bff-4960-a63b-584726e054eb","Type":"ContainerStarted","Data":"800b8cb38f4b7d785b3b4b233f4c566ab8dee9d0d28ca0fe5a43b1a54563e004"}
Jul 24 22:47:31.152116 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.152097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-48cpz" event={"ID":"7fb93a54-2070-48f9-9810-e3f4acb7250c","Type":"ContainerStarted","Data":"9780aabc98b73324ed93e8b2c9e308638c60782f831eea717b58504e99346b90"}
Jul 24 22:47:31.154178 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.154158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzzs7" event={"ID":"310b20a4-9c4c-44ec-bb15-f2c86b76e64a","Type":"ContainerStarted","Data":"5c1ccea452a5cf29d33f28d07882cf2849b30eabd49610de42259bf423b634e4"}
Jul 24 22:47:31.161835 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.161786    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7d8b58dcdd-7t9m9" podStartSLOduration=2.7151466969999998 podStartE2EDuration="9.161771536s" totalImagesPullingTime="6.446624839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:31.160990493 +0000 UTC m=+67.794269434" watchObservedRunningTime="2026-07-24 22:47:31.161771536 +0000 UTC m=+67.795050475"
Jul 24 22:47:31.178147 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.178035    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f685fb595-65fz9" podStartSLOduration=2.728621445 podStartE2EDuration="9.177988178s" totalImagesPullingTime="6.449366733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:31.176920312 +0000 UTC m=+67.810199253" watchObservedRunningTime="2026-07-24 22:47:31.177988178 +0000 UTC m=+67.811267110"
Jul 24 22:47:31.198862 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:31.198803    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jzzs7" podStartSLOduration=17.457856856 podStartE2EDuration="33.198785765s" totalImagesPullingTime="15.740928909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:31.198186205 +0000 UTC m=+67.831465146" watchObservedRunningTime="2026-07-24 22:47:31.198785765 +0000 UTC m=+67.832064706"
Jul 24 22:47:32.162663 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.162438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nspfm" event={"ID":"35f71564-7bff-4960-a63b-584726e054eb","Type":"ContainerStarted","Data":"444f0c310941a50b0743c0a1495a2f8f4ab6796ee8df660564cbd9127a736c65"}
Jul 24 22:47:32.162663 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.162507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nspfm" event={"ID":"35f71564-7bff-4960-a63b-584726e054eb","Type":"ContainerStarted","Data":"bd4e6a0fb42784ad2b88e299f756808063824f86ccca10cbad18ffe29afd0318"}
Jul 24 22:47:32.181479 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.181425    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nspfm" podStartSLOduration=68.095383089 podStartE2EDuration="1m9.181407171s" totalImagesPullingTime="1.086024082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:32.180880409 +0000 UTC m=+68.814159350" watchObservedRunningTime="2026-07-24 22:47:32.181407171 +0000 UTC m=+68.814686156"
Jul 24 22:47:32.262211 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.262177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:32.350367 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.350330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:32.437422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.437333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:32.514204 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.514167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:32.622877 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.622843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:32.777632 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.777545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:32.862884 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:32.862849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:33.008089 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:33.008033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:33.064931 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:33.064855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:33.238263 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:33.238224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:33.308925 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:33.308890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:33.626618 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:33.626581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:33.701383 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:33.701342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:33.976462 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:33.976371    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:34.076109 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:34.076079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:34.263776 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:34.263694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:34.418665 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:34.418626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:34.519732 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:34.519643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:35.489803 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:35.489777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:35.590688 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:35.590654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:36.177772 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:36.177740    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rv5zf" event={"ID":"303ae2ca-2b38-4094-9507-21820dc52eeb","Type":"ContainerStarted","Data":"5bf196c99cbc12d826f17b7bd10df96476eaa0cecf786e24f9dfcfa7a3f08f67"}
Jul 24 22:47:36.179204 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:36.179183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-48cpz" event={"ID":"7fb93a54-2070-48f9-9810-e3f4acb7250c","Type":"ContainerStarted","Data":"32d4d2fd6b11ae858c80212b70cccae5e0ac9da4d203aa3ef624360142e9171c"}
Jul 24 22:47:36.179337 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:36.179317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:47:36.192852 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:36.192799    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rv5zf" podStartSLOduration=68.690566244 podStartE2EDuration="1m13.192781767s" totalImagesPullingTime="4.502215523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:36.19210784 +0000 UTC m=+72.825386780" watchObservedRunningTime="2026-07-24 22:47:36.192781767 +0000 UTC m=+72.826060707"
Jul 24 22:47:36.207952 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:36.207904    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-48cpz" podStartSLOduration=68.713651543 podStartE2EDuration="1m13.207887929s" totalImagesPullingTime="4.494236386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:36.207363965 +0000 UTC m=+72.840642904" watchObservedRunningTime="2026-07-24 22:47:36.207887929 +0000 UTC m=+72.841166868"
Jul 24 22:47:36.271701 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:36.271665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:37.216692 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:37.216654    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:37.760912 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:37.760880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:37.997227 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:37.997198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:38.028450 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.028352    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q22z2"]
Jul 24 22:47:38.059742 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.059705    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.062715 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.062579    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 22:47:38.062862 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.062727    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jjfp8\""
Jul 24 22:47:38.062862 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.062728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 22:47:38.063795 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.063773    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 22:47:38.063920 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.063775    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 22:47:38.063920 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.063797    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 22:47:38.063920 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.063798    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 22:47:38.092098 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.092068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:38.110115 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-tls\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.110115 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9570cf0b-24f4-4b27-a89a-018d38a45a1d-metrics-client-ca\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.110275 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.110275 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.110275 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110214    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-wtmp\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.110275 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-root\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.110416 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-textfile\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.110416 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrnn\" (UniqueName: \"kubernetes.io/projected/9570cf0b-24f4-4b27-a89a-018d38a45a1d-kube-api-access-wlrnn\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.110416 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.110352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-sys\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.190798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.190764    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:38.211215 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-wtmp\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211215 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-root\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-textfile\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrnn\" (UniqueName: \"kubernetes.io/projected/9570cf0b-24f4-4b27-a89a-018d38a45a1d-kube-api-access-wlrnn\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-sys\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-tls\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-root\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9570cf0b-24f4-4b27-a89a-018d38a45a1d-metrics-client-ca\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-sys\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211368    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-wtmp\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211431 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211769 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211769 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211676    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-textfile\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.211988 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.211969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9570cf0b-24f4-4b27-a89a-018d38a45a1d-metrics-client-ca\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.212045 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.212012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-accelerators-collector-config\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.213732 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.213709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-tls\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.213851 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.213745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9570cf0b-24f4-4b27-a89a-018d38a45a1d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.219840 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.219819    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrnn\" (UniqueName: \"kubernetes.io/projected/9570cf0b-24f4-4b27-a89a-018d38a45a1d-kube-api-access-wlrnn\") pod \"node-exporter-q22z2\" (UID: \"9570cf0b-24f4-4b27-a89a-018d38a45a1d\") " pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.290329 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.290246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:38.371448 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.371414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q22z2"
Jul 24 22:47:38.381136 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:38.381108    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9570cf0b_24f4_4b27_a89a_018d38a45a1d.slice/crio-74f4dd090a9fd832c8c4ad422e8ca3621e3d8511f32c10790921adab66e62735 WatchSource:0}: Error finding container 74f4dd090a9fd832c8c4ad422e8ca3621e3d8511f32c10790921adab66e62735: Status 404 returned error can't find the container with id 74f4dd090a9fd832c8c4ad422e8ca3621e3d8511f32c10790921adab66e62735
Jul 24 22:47:38.390084 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.390053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:38.752297 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.752264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:38.993771 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:38.993738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:39.087314 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:39.087223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:39.190354 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:39.190322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q22z2" event={"ID":"9570cf0b-24f4-4b27-a89a-018d38a45a1d","Type":"ContainerStarted","Data":"74f4dd090a9fd832c8c4ad422e8ca3621e3d8511f32c10790921adab66e62735"}
Jul 24 22:47:39.310398 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:39.310365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:39.783637 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:39.783558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:39.883880 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:39.883850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:40.194597 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:40.194558    2563 generic.go:356] "Generic (PLEG): container finished" podID="9570cf0b-24f4-4b27-a89a-018d38a45a1d" containerID="716b2cbb8905d90ee253a37adc00ac0b1c45e8b46358f46bdaf42327d2817cb8" exitCode=0
Jul 24 22:47:40.194597 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:40.194602    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q22z2" event={"ID":"9570cf0b-24f4-4b27-a89a-018d38a45a1d","Type":"ContainerDied","Data":"716b2cbb8905d90ee253a37adc00ac0b1c45e8b46358f46bdaf42327d2817cb8"}
Jul 24 22:47:40.962817 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:40.962763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:41.066867 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.066837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:41.098826 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.098791    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7848bb556d-skdzx"]
Jul 24 22:47:41.101005 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.100988    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.104370 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.104343    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 24 22:47:41.104557 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.104508    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 24 22:47:41.104737 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.104715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 24 22:47:41.104859 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.104756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8nh2m\""
Jul 24 22:47:41.104992 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.104976    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 24 22:47:41.105299 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.105284    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 24 22:47:41.108032 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.108010    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b07heb1461mrp\""
Jul 24 22:47:41.141218 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.141175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7848bb556d-skdzx"]
Jul 24 22:47:41.201341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.201297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q22z2" event={"ID":"9570cf0b-24f4-4b27-a89a-018d38a45a1d","Type":"ContainerStarted","Data":"d20a3aa4e735f68c4d493b4602f784022d46e16dbbf3ff0dac53961555774e06"}
Jul 24 22:47:41.201568 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.201367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q22z2" event={"ID":"9570cf0b-24f4-4b27-a89a-018d38a45a1d","Type":"ContainerStarted","Data":"71e60ee6c149230fb40935cfb436edfc345a2a4edfef045e3124b754619d205e"}
Jul 24 22:47:41.222441 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.222320    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q22z2" podStartSLOduration=2.314520831 podStartE2EDuration="3.222297757s" totalImagesPullingTime="839.536588ms" totalInitContainerRuntime="68.240338ms" isStatefulPod=true podCreationTimestamp="2026-07-24 22:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:41.221356509 +0000 UTC m=+77.854635452" watchObservedRunningTime="2026-07-24 22:47:41.222297757 +0000 UTC m=+77.855576699"
Jul 24 22:47:41.235718 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.235665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.235897 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.235744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.235897 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.235797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6lq\" (UniqueName: \"kubernetes.io/projected/2f37c0e4-fe28-4382-9e8b-81667a365887-kube-api-access-8r6lq\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.235897 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.235822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.235897 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.235855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-tls\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.235897 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.235884    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f37c0e4-fe28-4382-9e8b-81667a365887-metrics-client-ca\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.236121 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.235915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.236121 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.235976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-grpc-tls\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.304183 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.304151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:41.336977 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.336945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6lq\" (UniqueName: \"kubernetes.io/projected/2f37c0e4-fe28-4382-9e8b-81667a365887-kube-api-access-8r6lq\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.337066 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.336984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.337174 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.337148    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-tls\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.337236 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.337211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f37c0e4-fe28-4382-9e8b-81667a365887-metrics-client-ca\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.337350 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.337325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.337483 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.337465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-grpc-tls\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.337573 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.337516    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.337626 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.337575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.338068 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.338044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f37c0e4-fe28-4382-9e8b-81667a365887-metrics-client-ca\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.339916 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.339883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.339916 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.339903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-tls\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.340116 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.340093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.340302 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.340280    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.340338 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.340330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-grpc-tls\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.340573 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.340556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2f37c0e4-fe28-4382-9e8b-81667a365887-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.346037 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.346011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6lq\" (UniqueName: \"kubernetes.io/projected/2f37c0e4-fe28-4382-9e8b-81667a365887-kube-api-access-8r6lq\") pod \"thanos-querier-7848bb556d-skdzx\" (UID: \"2f37c0e4-fe28-4382-9e8b-81667a365887\") " pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.362659 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.362627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:41.415838 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.415795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:41.579678 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:41.579648    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7848bb556d-skdzx"]
Jul 24 22:47:41.582582 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:41.582554    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f37c0e4_fe28_4382_9e8b_81667a365887.slice/crio-d8de74823618bfe1ebf7083a37214f4b82f86b9db9c4614a1b26fb058340f34a WatchSource:0}: Error finding container d8de74823618bfe1ebf7083a37214f4b82f86b9db9c4614a1b26fb058340f34a: Status 404 returned error can't find the container with id d8de74823618bfe1ebf7083a37214f4b82f86b9db9c4614a1b26fb058340f34a
Jul 24 22:47:42.205413 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.205370    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx" event={"ID":"2f37c0e4-fe28-4382-9e8b-81667a365887","Type":"ContainerStarted","Data":"d8de74823618bfe1ebf7083a37214f4b82f86b9db9c4614a1b26fb058340f34a"}
Jul 24 22:47:42.323011 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.322980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:42.426911 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.426876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:42.780169 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.780133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:42.794603 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.794570    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"]
Jul 24 22:47:42.797944 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.797916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:42.800636 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.800614    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 24 22:47:42.800782 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.800660    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gqwd8\""
Jul 24 22:47:42.808841 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.808781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"]
Jul 24 22:47:42.850742 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.850699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pkd6d\" (UID: \"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:42.850925 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.850754    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5hk2\" (UniqueName: \"kubernetes.io/projected/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-kube-api-access-j5hk2\") pod \"monitoring-plugin-68b7fc9574-pkd6d\" (UID: \"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:42.858911 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.858886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:42.952186 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.952151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pkd6d\" (UID: \"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:42.952393 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.952201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5hk2\" (UniqueName: \"kubernetes.io/projected/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-kube-api-access-j5hk2\") pod \"monitoring-plugin-68b7fc9574-pkd6d\" (UID: \"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:42.952393 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:42.952318    2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 24 22:47:42.952506 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:47:42.952394    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-monitoring-plugin-cert podName:ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf nodeName:}" failed. No retries permitted until 2026-07-24 22:47:43.452373017 +0000 UTC m=+80.085651951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-pkd6d" (UID: "ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf") : secret "monitoring-plugin-cert" not found
Jul 24 22:47:42.963122 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:42.963087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5hk2\" (UniqueName: \"kubernetes.io/projected/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-kube-api-access-j5hk2\") pod \"monitoring-plugin-68b7fc9574-pkd6d\" (UID: \"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:43.369640 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:43.369604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:43.457003 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:43.456970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pkd6d\" (UID: \"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:43.459788 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:43.459759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pkd6d\" (UID: \"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:43.710297 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:43.710273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:43.864736 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:43.864699    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"]
Jul 24 22:47:43.868947 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:43.868913    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb3f17a_e8eb_4013_99c6_bf2ac5a82baf.slice/crio-22f0c1e9221e6041120a439eb0229d9b75999e3c3267c1cf7fc346ab79a7b35a WatchSource:0}: Error finding container 22f0c1e9221e6041120a439eb0229d9b75999e3c3267c1cf7fc346ab79a7b35a: Status 404 returned error can't find the container with id 22f0c1e9221e6041120a439eb0229d9b75999e3c3267c1cf7fc346ab79a7b35a
Jul 24 22:47:44.218730 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:44.218691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx" event={"ID":"2f37c0e4-fe28-4382-9e8b-81667a365887","Type":"ContainerStarted","Data":"1456e386b232790cf7e71cff504f2ccc51dbfd18776f185aa31f1d241fefd164"}
Jul 24 22:47:44.218930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:44.218770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx" event={"ID":"2f37c0e4-fe28-4382-9e8b-81667a365887","Type":"ContainerStarted","Data":"26fe2b0939c18fb9c9f8c013856620132d899b29d3320b0a88c52641b840b12f"}
Jul 24 22:47:44.218930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:44.218809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx" event={"ID":"2f37c0e4-fe28-4382-9e8b-81667a365887","Type":"ContainerStarted","Data":"cb8573a4da39528b24a84740232a1bb8891303be55eb6d7e92549e7508784bf5"}
Jul 24 22:47:44.219766 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:44.219741    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d" event={"ID":"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf","Type":"ContainerStarted","Data":"22f0c1e9221e6041120a439eb0229d9b75999e3c3267c1cf7fc346ab79a7b35a"}
Jul 24 22:47:44.260496 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:44.260463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:45.228020 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.227980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx" event={"ID":"2f37c0e4-fe28-4382-9e8b-81667a365887","Type":"ContainerStarted","Data":"d879b15abdab83f3c3170658016f69575a0b5fd0fd751c9fbf2190b98982af04"}
Jul 24 22:47:45.228506 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.228063    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx" event={"ID":"2f37c0e4-fe28-4382-9e8b-81667a365887","Type":"ContainerStarted","Data":"5b0c99a1f10b2d00291818da2b470e0fef45b90e9038902de6398f702ecafda9"}
Jul 24 22:47:45.228506 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.228108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx" event={"ID":"2f37c0e4-fe28-4382-9e8b-81667a365887","Type":"ContainerStarted","Data":"0487930daf33307c2bd21c6da7c567195014dcecabca17cdaa0b1c57f3f18f1a"}
Jul 24 22:47:45.228506 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.228151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:45.255906 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.254682    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx" podStartSLOduration=1.229595286 podStartE2EDuration="4.254657093s" totalImagesPullingTime="3.025061807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:45.254463183 +0000 UTC m=+81.887742137" watchObservedRunningTime="2026-07-24 22:47:45.254657093 +0000 UTC m=+81.887936033"
Jul 24 22:47:45.539216 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.539130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:45.591006 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.590977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:45.711389 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.711357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:45.812660 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:45.812582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:46.233323 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:46.233284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d" event={"ID":"ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf","Type":"ContainerStarted","Data":"8e67fc31c2ef0e7b79986281a70220e5ad6b65824aa242c580e802e353599703"}
Jul 24 22:47:46.233798 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:46.233769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:46.238689 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:46.238660    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d"
Jul 24 22:47:46.250509 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:46.250358    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pkd6d" podStartSLOduration=2.713659656 podStartE2EDuration="4.25033603s" totalImagesPullingTime="1.536676374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:46.250057564 +0000 UTC m=+82.883336504" watchObservedRunningTime="2026-07-24 22:47:46.25033603 +0000 UTC m=+82.883614971"
Jul 24 22:47:46.438442 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:46.438405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:46.540791 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:46.540704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:46.638484 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:46.638450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:48.842168 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.842133    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b8cd54c99-wjr57"]
Jul 24 22:47:48.849378 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.849351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:48.851997 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.851973    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 24 22:47:48.853430 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.853406    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 24 22:47:48.853560 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.853434    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 24 22:47:48.853560 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.853441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 24 22:47:48.853560 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.853473    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j5s8t\""
Jul 24 22:47:48.853560 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.853501    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 24 22:47:48.853560 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.853515    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 24 22:47:48.854302 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.854282    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 24 22:47:48.855284 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.855262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b8cd54c99-wjr57"]
Jul 24 22:47:48.859286 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:48.859264    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 24 22:47:49.003054 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.003015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-oauth-config\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.003054 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.003056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-serving-cert\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.003294 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.003087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-config\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.003294 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.003144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-service-ca\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.003294 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.003195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-oauth-serving-cert\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.003294 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.003227    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-trusted-ca-bundle\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.003294 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.003253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zscd\" (UniqueName: \"kubernetes.io/projected/d5efd0dd-391a-4660-b546-8fcb6f444e20-kube-api-access-4zscd\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.104341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.104247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-serving-cert\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.104341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.104288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-config\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.104341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.104311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-service-ca\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.104341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.104337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-oauth-serving-cert\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.104713 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.104372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-trusted-ca-bundle\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.104713 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.104407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zscd\" (UniqueName: \"kubernetes.io/projected/d5efd0dd-391a-4660-b546-8fcb6f444e20-kube-api-access-4zscd\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.104713 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.104450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-oauth-config\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.105150 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.105125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-oauth-serving-cert\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.105246 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.105125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-config\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.105246 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.105200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-service-ca\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.105397 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.105377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-trusted-ca-bundle\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.106921 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.106899    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-serving-cert\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.106921 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.106909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-oauth-config\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.113384 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.113357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zscd\" (UniqueName: \"kubernetes.io/projected/d5efd0dd-391a-4660-b546-8fcb6f444e20-kube-api-access-4zscd\") pod \"console-5b8cd54c99-wjr57\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") " pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.160149 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.160109    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:49.333041 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.333006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b8cd54c99-wjr57"]
Jul 24 22:47:49.335929 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:47:49.335885    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5efd0dd_391a_4660_b546_8fcb6f444e20.slice/crio-8fa986afc7eb057efda5aa6985b51f432d28c404c9b0f1730279c7c176b006ba WatchSource:0}: Error finding container 8fa986afc7eb057efda5aa6985b51f432d28c404c9b0f1730279c7c176b006ba: Status 404 returned error can't find the container with id 8fa986afc7eb057efda5aa6985b51f432d28c404c9b0f1730279c7c176b006ba
Jul 24 22:47:49.830561 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.830507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:49.931318 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:49.931284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:50.256127 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:50.254151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8cd54c99-wjr57" event={"ID":"d5efd0dd-391a-4660-b546-8fcb6f444e20","Type":"ContainerStarted","Data":"8fa986afc7eb057efda5aa6985b51f432d28c404c9b0f1730279c7c176b006ba"}
Jul 24 22:47:50.490485 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:50.490449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:50.565301 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:50.565216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:51.106119 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:51.106084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:51.207269 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:51.207235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:51.240765 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:51.240736    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7848bb556d-skdzx"
Jul 24 22:47:51.633955 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:51.633921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:51.735186 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:51.735133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:53.267109 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:53.267078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8cd54c99-wjr57" event={"ID":"d5efd0dd-391a-4660-b546-8fcb6f444e20","Type":"ContainerStarted","Data":"8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920"}
Jul 24 22:47:53.286624 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:53.286570    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b8cd54c99-wjr57" podStartSLOduration=1.9235779480000001 podStartE2EDuration="5.286553763s" totalImagesPullingTime="3.362975815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:53.285209242 +0000 UTC m=+89.918488183" watchObservedRunningTime="2026-07-24 22:47:53.286553763 +0000 UTC m=+89.919832703"
Jul 24 22:47:54.779659 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:54.779624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:54.877746 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:54.877718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:55.516030 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:55.515998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:55.618996 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:55.618971    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:56.090139 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:56.090111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:56.194134 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:56.194107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:58.439380 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:58.439349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:58.500361 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:58.500328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:59.160880 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:59.160844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:59.161060 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:59.160912    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:59.165729 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:59.165705    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:59.293461 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:59.293432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:47:59.560807 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:59.560720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:47:59.616405 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:47:59.616376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:48:00.674811 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:48:00.674781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:48:00.762086 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:48:00.762056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:48:07.185055 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:48:07.185023    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-48cpz"
Jul 24 22:48:09.765498 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:48:09.765465    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:48:09.819152 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:48:09.819129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:48:22.188011 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:48:22.187970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:48:57.279939 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:48:57.279903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:48:57.431875 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:48:57.431841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:07.647751 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.647710    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-776875b558-shs8r"]
Jul 24 22:49:07.649834 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.649816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.669319 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.669284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776875b558-shs8r"]
Jul 24 22:49:07.696295 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.696252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:07.703381 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.703351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-oauth-serving-cert\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.703494 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.703385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-service-ca\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.703494 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.703414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-config\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.703610 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.703502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tff\" (UniqueName: \"kubernetes.io/projected/d74b3b9f-bd99-44e3-9606-c0da9c59638f-kube-api-access-c9tff\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.703610 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.703548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-oauth-config\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.703610 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.703565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-serving-cert\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.703610 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.703583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-trusted-ca-bundle\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.804460 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.804412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tff\" (UniqueName: \"kubernetes.io/projected/d74b3b9f-bd99-44e3-9606-c0da9c59638f-kube-api-access-c9tff\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.804460 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.804452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-oauth-config\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.804460 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.804471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-serving-cert\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.804810 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.804489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-trusted-ca-bundle\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.804810 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.804574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-oauth-serving-cert\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.804810 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.804614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-service-ca\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.804810 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.804663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-config\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.805495 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.805457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-oauth-serving-cert\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.805653 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.805464    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-config\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.805653 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.805500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-trusted-ca-bundle\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.805653 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.805537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-service-ca\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.807126 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.807104    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-oauth-config\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.807221 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.807202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-serving-cert\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.815053 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.815021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tff\" (UniqueName: \"kubernetes.io/projected/d74b3b9f-bd99-44e3-9606-c0da9c59638f-kube-api-access-c9tff\") pod \"console-776875b558-shs8r\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") " pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:07.845700 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.845668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:07.901000 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.900922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:07.959830 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:07.959795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:08.096842 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:08.096817    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-776875b558-shs8r"]
Jul 24 22:49:08.099466 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:49:08.099439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74b3b9f_bd99_44e3_9606_c0da9c59638f.slice/crio-5df470d61883b62d43269ddb254df0a4b7d9670bc567cd0c8618d2e4234ff9c9 WatchSource:0}: Error finding container 5df470d61883b62d43269ddb254df0a4b7d9670bc567cd0c8618d2e4234ff9c9: Status 404 returned error can't find the container with id 5df470d61883b62d43269ddb254df0a4b7d9670bc567cd0c8618d2e4234ff9c9
Jul 24 22:49:08.447903 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:08.447789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:08.536654 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:08.536617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776875b558-shs8r" event={"ID":"d74b3b9f-bd99-44e3-9606-c0da9c59638f","Type":"ContainerStarted","Data":"09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e"}
Jul 24 22:49:08.536830 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:08.536667    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776875b558-shs8r" event={"ID":"d74b3b9f-bd99-44e3-9606-c0da9c59638f","Type":"ContainerStarted","Data":"5df470d61883b62d43269ddb254df0a4b7d9670bc567cd0c8618d2e4234ff9c9"}
Jul 24 22:49:08.555307 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:08.555254    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-776875b558-shs8r" podStartSLOduration=1.555234931 podStartE2EDuration="1.555234931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:49:08.554271376 +0000 UTC m=+165.187550317" watchObservedRunningTime="2026-07-24 22:49:08.555234931 +0000 UTC m=+165.188513870"
Jul 24 22:49:17.960816 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:17.960771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:17.961236 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:17.960845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:17.965850 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:17.965822    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:18.575672 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:18.575642    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:49:18.619934 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:18.619894    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b8cd54c99-wjr57"]
Jul 24 22:49:18.717911 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:18.717877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:18.818648 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:18.818618    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:18.870054 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:18.870022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:43.641041 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.640995    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b8cd54c99-wjr57" podUID="d5efd0dd-391a-4660-b546-8fcb6f444e20" containerName="console" containerID="cri-o://8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920" gracePeriod=15
Jul 24 22:49:43.888280 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.888251    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:49:43.982846 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.982752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-trusted-ca-bundle\") pod \"d5efd0dd-391a-4660-b546-8fcb6f444e20\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") "
Jul 24 22:49:43.982846 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.982802    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-oauth-serving-cert\") pod \"d5efd0dd-391a-4660-b546-8fcb6f444e20\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") "
Jul 24 22:49:43.982846 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.982827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-config\") pod \"d5efd0dd-391a-4660-b546-8fcb6f444e20\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") "
Jul 24 22:49:43.983116 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.982865    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-service-ca\") pod \"d5efd0dd-391a-4660-b546-8fcb6f444e20\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") "
Jul 24 22:49:43.983116 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.982965    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-oauth-config\") pod \"d5efd0dd-391a-4660-b546-8fcb6f444e20\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") "
Jul 24 22:49:43.983116 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.983001    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zscd\" (UniqueName: \"kubernetes.io/projected/d5efd0dd-391a-4660-b546-8fcb6f444e20-kube-api-access-4zscd\") pod \"d5efd0dd-391a-4660-b546-8fcb6f444e20\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") "
Jul 24 22:49:43.983116 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.983031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-serving-cert\") pod \"d5efd0dd-391a-4660-b546-8fcb6f444e20\" (UID: \"d5efd0dd-391a-4660-b546-8fcb6f444e20\") "
Jul 24 22:49:43.983361 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.983261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-service-ca" (OuterVolumeSpecName: "service-ca") pod "d5efd0dd-391a-4660-b546-8fcb6f444e20" (UID: "d5efd0dd-391a-4660-b546-8fcb6f444e20"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:49:43.983361 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.983307    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-config" (OuterVolumeSpecName: "console-config") pod "d5efd0dd-391a-4660-b546-8fcb6f444e20" (UID: "d5efd0dd-391a-4660-b546-8fcb6f444e20"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:49:43.983361 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.983331    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d5efd0dd-391a-4660-b546-8fcb6f444e20" (UID: "d5efd0dd-391a-4660-b546-8fcb6f444e20"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:49:43.983503 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.983374    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:49:43.983503 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.983422    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d5efd0dd-391a-4660-b546-8fcb6f444e20" (UID: "d5efd0dd-391a-4660-b546-8fcb6f444e20"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:49:43.985341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.985311    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5efd0dd-391a-4660-b546-8fcb6f444e20-kube-api-access-4zscd" (OuterVolumeSpecName: "kube-api-access-4zscd") pod "d5efd0dd-391a-4660-b546-8fcb6f444e20" (UID: "d5efd0dd-391a-4660-b546-8fcb6f444e20"). InnerVolumeSpecName "kube-api-access-4zscd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:49:43.985341 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.985314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d5efd0dd-391a-4660-b546-8fcb6f444e20" (UID: "d5efd0dd-391a-4660-b546-8fcb6f444e20"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:49:43.985505 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:43.985364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d5efd0dd-391a-4660-b546-8fcb6f444e20" (UID: "d5efd0dd-391a-4660-b546-8fcb6f444e20"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:49:44.083986 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.083946    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-trusted-ca-bundle\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:49:44.083986 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.083981    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-oauth-serving-cert\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:49:44.083986 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.083992    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-config\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:49:44.084234 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.084001    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-oauth-config\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:49:44.084234 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.084010    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zscd\" (UniqueName: \"kubernetes.io/projected/d5efd0dd-391a-4660-b546-8fcb6f444e20-kube-api-access-4zscd\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:49:44.084234 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.084019    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5efd0dd-391a-4660-b546-8fcb6f444e20-console-serving-cert\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:49:44.665302 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.665266    2563 generic.go:356] "Generic (PLEG): container finished" podID="d5efd0dd-391a-4660-b546-8fcb6f444e20" containerID="8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920" exitCode=0
Jul 24 22:49:44.665846 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.665314    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8cd54c99-wjr57"
Jul 24 22:49:44.665846 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.665355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8cd54c99-wjr57" event={"ID":"d5efd0dd-391a-4660-b546-8fcb6f444e20","Type":"ContainerDied","Data":"8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920"}
Jul 24 22:49:44.665846 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.665413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8cd54c99-wjr57" event={"ID":"d5efd0dd-391a-4660-b546-8fcb6f444e20","Type":"ContainerDied","Data":"8fa986afc7eb057efda5aa6985b51f432d28c404c9b0f1730279c7c176b006ba"}
Jul 24 22:49:44.665846 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.665436    2563 scope.go:117] "RemoveContainer" containerID="8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920"
Jul 24 22:49:44.675777 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.675756    2563 scope.go:117] "RemoveContainer" containerID="8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920"
Jul 24 22:49:44.676207 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:49:44.676183    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920\": container with ID starting with 8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920 not found: ID does not exist" containerID="8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920"
Jul 24 22:49:44.676255 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.676218    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920"} err="failed to get container status \"8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920\": rpc error: code = NotFound desc = could not find container \"8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920\": container with ID starting with 8dc47988ba63508b55854cdfa09c2b639d51421d2d003937a9294ffa3062a920 not found: ID does not exist"
Jul 24 22:49:44.686360 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.686264    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b8cd54c99-wjr57"]
Jul 24 22:49:44.688359 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:44.688329    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b8cd54c99-wjr57"]
Jul 24 22:49:45.840438 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:45.840407    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5efd0dd-391a-4660-b546-8fcb6f444e20" path="/var/lib/kubelet/pods/d5efd0dd-391a-4660-b546-8fcb6f444e20/volumes"
Jul 24 22:49:47.637999 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:47.637936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:49:47.742739 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:49:47.742704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:12.475974 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:12.475942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:12.524103 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:12.524069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:22.949549 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:22.949492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:24.494506 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:24.494472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:24.595429 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:24.595398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:30.097157 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.097120    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fd5b5b74b-v75cs"]
Jul 24 22:50:30.097647 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.097395    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5efd0dd-391a-4660-b546-8fcb6f444e20" containerName="console"
Jul 24 22:50:30.097647 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.097408    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5efd0dd-391a-4660-b546-8fcb6f444e20" containerName="console"
Jul 24 22:50:30.097647 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.097461    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5efd0dd-391a-4660-b546-8fcb6f444e20" containerName="console"
Jul 24 22:50:30.100528 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.100499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.111668 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.111635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd5b5b74b-v75cs"]
Jul 24 22:50:30.154612 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.154569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:30.206644 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.206616    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:30.212840 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.212808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-serving-cert\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.212920 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.212848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-trusted-ca-bundle\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.212920 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.212885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-service-ca\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.212985 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.212930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-oauth-config\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.212985 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.212948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-oauth-serving-cert\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.213050 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.213003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrb4\" (UniqueName: \"kubernetes.io/projected/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-kube-api-access-7mrb4\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.213050 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.213024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-config\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.301854 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.301824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:30.313936 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.313893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-oauth-config\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.313936 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.313933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-oauth-serving-cert\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.314163 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.313972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrb4\" (UniqueName: \"kubernetes.io/projected/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-kube-api-access-7mrb4\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.314163 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.313994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-config\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.314163 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.314022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-serving-cert\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.314163 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.314038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-trusted-ca-bundle\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.314163 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.314063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-service-ca\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.314795 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.314766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-oauth-serving-cert\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.314900 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.314799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-config\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.314900 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.314860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-service-ca\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.315014 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.314918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-trusted-ca-bundle\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.316473 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.316448    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-oauth-config\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.316621 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.316597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-serving-cert\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.323391 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.323358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrb4\" (UniqueName: \"kubernetes.io/projected/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-kube-api-access-7mrb4\") pod \"console-6fd5b5b74b-v75cs\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") " pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.398809 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.398725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:30.409692 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.409656    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:30.543683 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.543651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fd5b5b74b-v75cs"]
Jul 24 22:50:30.547175 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:50:30.547142    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1eb538_ce1c_42e4_9ebe_be610a91e1ab.slice/crio-a922901218a6398f1c81cf292914d902e2db98fccd7edd9903a2f90fddbed365 WatchSource:0}: Error finding container a922901218a6398f1c81cf292914d902e2db98fccd7edd9903a2f90fddbed365: Status 404 returned error can't find the container with id a922901218a6398f1c81cf292914d902e2db98fccd7edd9903a2f90fddbed365
Jul 24 22:50:30.826869 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.826782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd5b5b74b-v75cs" event={"ID":"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab","Type":"ContainerStarted","Data":"abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81"}
Jul 24 22:50:30.826869 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.826831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd5b5b74b-v75cs" event={"ID":"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab","Type":"ContainerStarted","Data":"a922901218a6398f1c81cf292914d902e2db98fccd7edd9903a2f90fddbed365"}
Jul 24 22:50:30.865537 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.865464    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fd5b5b74b-v75cs" podStartSLOduration=0.865447849 podStartE2EDuration="865.447849ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:50:30.865392661 +0000 UTC m=+247.498671614" watchObservedRunningTime="2026-07-24 22:50:30.865447849 +0000 UTC m=+247.498726788"
Jul 24 22:50:30.917416 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.917364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:30.986796 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:30.986766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:39.775927 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:39.775891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:39.874012 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:39.873985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:40.410082 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:40.410027    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:40.410082 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:40.410101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:40.414926 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:40.414904    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:40.863882 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:40.863856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:50:40.910491 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:40.910454    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776875b558-shs8r"]
Jul 24 22:50:41.011295 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:41.011247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:41.112774 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:41.112741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:41.212786 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:41.212698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:50:58.289060 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:50:58.288973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:05.934829 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:05.934771    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-776875b558-shs8r" podUID="d74b3b9f-bd99-44e3-9606-c0da9c59638f" containerName="console" containerID="cri-o://09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e" gracePeriod=15
Jul 24 22:51:06.178690 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.178664    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:51:06.292065 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.291977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-oauth-serving-cert\") pod \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") "
Jul 24 22:51:06.292065 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-service-ca\") pod \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") "
Jul 24 22:51:06.292304 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9tff\" (UniqueName: \"kubernetes.io/projected/d74b3b9f-bd99-44e3-9606-c0da9c59638f-kube-api-access-c9tff\") pod \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") "
Jul 24 22:51:06.292304 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292102    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-trusted-ca-bundle\") pod \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") "
Jul 24 22:51:06.292304 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292248    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-oauth-config\") pod \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") "
Jul 24 22:51:06.292304 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-serving-cert\") pod \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") "
Jul 24 22:51:06.292550 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-config\") pod \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\" (UID: \"d74b3b9f-bd99-44e3-9606-c0da9c59638f\") "
Jul 24 22:51:06.292550 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d74b3b9f-bd99-44e3-9606-c0da9c59638f" (UID: "d74b3b9f-bd99-44e3-9606-c0da9c59638f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:51:06.292550 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292457    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-service-ca" (OuterVolumeSpecName: "service-ca") pod "d74b3b9f-bd99-44e3-9606-c0da9c59638f" (UID: "d74b3b9f-bd99-44e3-9606-c0da9c59638f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:51:06.292550 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292490    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d74b3b9f-bd99-44e3-9606-c0da9c59638f" (UID: "d74b3b9f-bd99-44e3-9606-c0da9c59638f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:51:06.292752 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292610    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-oauth-serving-cert\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:51:06.292752 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292727    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:51:06.292752 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292741    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-trusted-ca-bundle\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:51:06.292907 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.292784    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-config" (OuterVolumeSpecName: "console-config") pod "d74b3b9f-bd99-44e3-9606-c0da9c59638f" (UID: "d74b3b9f-bd99-44e3-9606-c0da9c59638f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:51:06.294363 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.294338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d74b3b9f-bd99-44e3-9606-c0da9c59638f" (UID: "d74b3b9f-bd99-44e3-9606-c0da9c59638f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:51:06.294478 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.294381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74b3b9f-bd99-44e3-9606-c0da9c59638f-kube-api-access-c9tff" (OuterVolumeSpecName: "kube-api-access-c9tff") pod "d74b3b9f-bd99-44e3-9606-c0da9c59638f" (UID: "d74b3b9f-bd99-44e3-9606-c0da9c59638f"). InnerVolumeSpecName "kube-api-access-c9tff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:51:06.294478 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.294421    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d74b3b9f-bd99-44e3-9606-c0da9c59638f" (UID: "d74b3b9f-bd99-44e3-9606-c0da9c59638f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:51:06.394028 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.393987    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tff\" (UniqueName: \"kubernetes.io/projected/d74b3b9f-bd99-44e3-9606-c0da9c59638f-kube-api-access-c9tff\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:51:06.394028 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.394022    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-oauth-config\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:51:06.394028 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.394031    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-serving-cert\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:51:06.394262 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.394041    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d74b3b9f-bd99-44e3-9606-c0da9c59638f-console-config\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:51:06.949807 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.949774    2563 generic.go:356] "Generic (PLEG): container finished" podID="d74b3b9f-bd99-44e3-9606-c0da9c59638f" containerID="09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e" exitCode=0
Jul 24 22:51:06.950225 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.949819    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-776875b558-shs8r"
Jul 24 22:51:06.950225 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.949837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776875b558-shs8r" event={"ID":"d74b3b9f-bd99-44e3-9606-c0da9c59638f","Type":"ContainerDied","Data":"09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e"}
Jul 24 22:51:06.950225 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.949877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-776875b558-shs8r" event={"ID":"d74b3b9f-bd99-44e3-9606-c0da9c59638f","Type":"ContainerDied","Data":"5df470d61883b62d43269ddb254df0a4b7d9670bc567cd0c8618d2e4234ff9c9"}
Jul 24 22:51:06.950225 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.949896    2563 scope.go:117] "RemoveContainer" containerID="09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e"
Jul 24 22:51:06.958589 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.958566    2563 scope.go:117] "RemoveContainer" containerID="09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e"
Jul 24 22:51:06.958893 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:51:06.958875    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e\": container with ID starting with 09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e not found: ID does not exist" containerID="09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e"
Jul 24 22:51:06.958939 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.958902    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e"} err="failed to get container status \"09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e\": rpc error: code = NotFound desc = could not find container \"09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e\": container with ID starting with 09989c202de336b51070df1d9b571a7230d2d49cf4733cc52c083dc10b519e8e not found: ID does not exist"
Jul 24 22:51:06.971502 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.971475    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-776875b558-shs8r"]
Jul 24 22:51:06.978444 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:06.978420    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-776875b558-shs8r"]
Jul 24 22:51:07.839783 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:07.839739    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d74b3b9f-bd99-44e3-9606-c0da9c59638f" path="/var/lib/kubelet/pods/d74b3b9f-bd99-44e3-9606-c0da9c59638f/volumes"
Jul 24 22:51:19.133508 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:19.133473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:20.435427 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:20.435398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:23.713343 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:23.713313    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 22:51:23.717661 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:23.717622    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 22:51:23.717932 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:23.717913    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 22:51:45.806066 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:45.806029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:46.608643 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:46.608603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:49.424768 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:49.424734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:49.523279 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:49.523249    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:50.789905 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:50.789873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:50.904467 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:50.904433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:51.426871 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:51.426841    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:51.534260 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:51.534228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:53.674174 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:53.674149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:53.831162 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:53.831132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:54.174601 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.174572    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"]
Jul 24 22:51:54.174837 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.174826    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74b3b9f-bd99-44e3-9606-c0da9c59638f" containerName="console"
Jul 24 22:51:54.174883 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.174838    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74b3b9f-bd99-44e3-9606-c0da9c59638f" containerName="console"
Jul 24 22:51:54.174917 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.174891    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74b3b9f-bd99-44e3-9606-c0da9c59638f" containerName="console"
Jul 24 22:51:54.177685 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.177669    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"
Jul 24 22:51:54.180486 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.180463    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 24 22:51:54.181442 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.181430    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 24 22:51:54.181493 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.181463    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-dlmcp\""
Jul 24 22:51:54.185964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.185942    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"]
Jul 24 22:51:54.236020 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.235981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/946eac87-7fe3-4f09-a27b-97b36e36bef6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jx4j4\" (UID: \"946eac87-7fe3-4f09-a27b-97b36e36bef6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"
Jul 24 22:51:54.236191 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.236029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wph9w\" (UniqueName: \"kubernetes.io/projected/946eac87-7fe3-4f09-a27b-97b36e36bef6-kube-api-access-wph9w\") pod \"cert-manager-cainjector-559cdd8c87-jx4j4\" (UID: \"946eac87-7fe3-4f09-a27b-97b36e36bef6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"
Jul 24 22:51:54.336371 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.336331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/946eac87-7fe3-4f09-a27b-97b36e36bef6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jx4j4\" (UID: \"946eac87-7fe3-4f09-a27b-97b36e36bef6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"
Jul 24 22:51:54.336551 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.336385    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wph9w\" (UniqueName: \"kubernetes.io/projected/946eac87-7fe3-4f09-a27b-97b36e36bef6-kube-api-access-wph9w\") pod \"cert-manager-cainjector-559cdd8c87-jx4j4\" (UID: \"946eac87-7fe3-4f09-a27b-97b36e36bef6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"
Jul 24 22:51:54.345039 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.345001    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/946eac87-7fe3-4f09-a27b-97b36e36bef6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jx4j4\" (UID: \"946eac87-7fe3-4f09-a27b-97b36e36bef6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"
Jul 24 22:51:54.345181 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.345059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wph9w\" (UniqueName: \"kubernetes.io/projected/946eac87-7fe3-4f09-a27b-97b36e36bef6-kube-api-access-wph9w\") pod \"cert-manager-cainjector-559cdd8c87-jx4j4\" (UID: \"946eac87-7fe3-4f09-a27b-97b36e36bef6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"
Jul 24 22:51:54.487958 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.487863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"
Jul 24 22:51:54.618399 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.618272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4"]
Jul 24 22:51:54.621123 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:51:54.621094    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod946eac87_7fe3_4f09_a27b_97b36e36bef6.slice/crio-0902cf2d51ff56bddc408cf4aaf9f5f99a525fc657f0896fd4b6e84d72849a16 WatchSource:0}: Error finding container 0902cf2d51ff56bddc408cf4aaf9f5f99a525fc657f0896fd4b6e84d72849a16: Status 404 returned error can't find the container with id 0902cf2d51ff56bddc408cf4aaf9f5f99a525fc657f0896fd4b6e84d72849a16
Jul 24 22:51:54.623385 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:54.623367    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:51:55.120286 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:55.120245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4" event={"ID":"946eac87-7fe3-4f09-a27b-97b36e36bef6","Type":"ContainerStarted","Data":"0902cf2d51ff56bddc408cf4aaf9f5f99a525fc657f0896fd4b6e84d72849a16"}
Jul 24 22:51:56.223306 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:56.223260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:56.322413 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:56.322379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:58.132263 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:58.132225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4" event={"ID":"946eac87-7fe3-4f09-a27b-97b36e36bef6","Type":"ContainerStarted","Data":"b40ef76af4f3fe7c49c6bbe6295b92f31cdf1f72399d2f2c605760d026957a9d"}
Jul 24 22:51:58.149126 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:58.149074    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jx4j4" podStartSLOduration=1.250561996 podStartE2EDuration="4.149055118s" totalImagesPullingTime="2.898493122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:51:58.147510525 +0000 UTC m=+334.780789479" watchObservedRunningTime="2026-07-24 22:51:58.149055118 +0000 UTC m=+334.782334060"
Jul 24 22:51:58.300766 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:58.300729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:58.401739 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:58.401629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:59.835485 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:59.835447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:51:59.936186 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:51:59.936154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:01.850112 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:01.850081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:02.318298 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.318211    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hz6cq"]
Jul 24 22:52:02.321336 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.321312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hz6cq"
Jul 24 22:52:02.323917 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.323896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-snwdt\""
Jul 24 22:52:02.330989 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.330962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hz6cq"]
Jul 24 22:52:02.400143 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.400110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpdsg\" (UniqueName: \"kubernetes.io/projected/de3b8e85-f687-453c-8281-1bfc0aad1f16-kube-api-access-hpdsg\") pod \"cert-manager-55b4d954d5-hz6cq\" (UID: \"de3b8e85-f687-453c-8281-1bfc0aad1f16\") " pod="cert-manager/cert-manager-55b4d954d5-hz6cq"
Jul 24 22:52:02.400143 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.400154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de3b8e85-f687-453c-8281-1bfc0aad1f16-bound-sa-token\") pod \"cert-manager-55b4d954d5-hz6cq\" (UID: \"de3b8e85-f687-453c-8281-1bfc0aad1f16\") " pod="cert-manager/cert-manager-55b4d954d5-hz6cq"
Jul 24 22:52:02.501058 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.501018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpdsg\" (UniqueName: \"kubernetes.io/projected/de3b8e85-f687-453c-8281-1bfc0aad1f16-kube-api-access-hpdsg\") pod \"cert-manager-55b4d954d5-hz6cq\" (UID: \"de3b8e85-f687-453c-8281-1bfc0aad1f16\") " pod="cert-manager/cert-manager-55b4d954d5-hz6cq"
Jul 24 22:52:02.501230 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.501067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de3b8e85-f687-453c-8281-1bfc0aad1f16-bound-sa-token\") pod \"cert-manager-55b4d954d5-hz6cq\" (UID: \"de3b8e85-f687-453c-8281-1bfc0aad1f16\") " pod="cert-manager/cert-manager-55b4d954d5-hz6cq"
Jul 24 22:52:02.509562 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.509504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/de3b8e85-f687-453c-8281-1bfc0aad1f16-bound-sa-token\") pod \"cert-manager-55b4d954d5-hz6cq\" (UID: \"de3b8e85-f687-453c-8281-1bfc0aad1f16\") " pod="cert-manager/cert-manager-55b4d954d5-hz6cq"
Jul 24 22:52:02.509714 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.509573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpdsg\" (UniqueName: \"kubernetes.io/projected/de3b8e85-f687-453c-8281-1bfc0aad1f16-kube-api-access-hpdsg\") pod \"cert-manager-55b4d954d5-hz6cq\" (UID: \"de3b8e85-f687-453c-8281-1bfc0aad1f16\") " pod="cert-manager/cert-manager-55b4d954d5-hz6cq"
Jul 24 22:52:02.630907 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.630861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hz6cq"
Jul 24 22:52:02.766323 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:02.766164    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hz6cq"]
Jul 24 22:52:02.769034 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:52:02.768999    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde3b8e85_f687_453c_8281_1bfc0aad1f16.slice/crio-f863dd0e8525583e3255b58c902bbc7ff11fca45acc20701bac741824bea683e WatchSource:0}: Error finding container f863dd0e8525583e3255b58c902bbc7ff11fca45acc20701bac741824bea683e: Status 404 returned error can't find the container with id f863dd0e8525583e3255b58c902bbc7ff11fca45acc20701bac741824bea683e
Jul 24 22:52:03.151419 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:03.151380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hz6cq" event={"ID":"de3b8e85-f687-453c-8281-1bfc0aad1f16","Type":"ContainerStarted","Data":"2e9904f54e9280b3fe86e0eaff8e6eb6f96ca2e691a35aa7ac92c456305c926c"}
Jul 24 22:52:03.151419 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:03.151426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hz6cq" event={"ID":"de3b8e85-f687-453c-8281-1bfc0aad1f16","Type":"ContainerStarted","Data":"f863dd0e8525583e3255b58c902bbc7ff11fca45acc20701bac741824bea683e"}
Jul 24 22:52:03.166224 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:03.166169    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-hz6cq" podStartSLOduration=1.166151501 podStartE2EDuration="1.166151501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:03.165707173 +0000 UTC m=+339.798986114" watchObservedRunningTime="2026-07-24 22:52:03.166151501 +0000 UTC m=+339.799430441"
Jul 24 22:52:06.203810 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:06.203776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:06.301513 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:06.301477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:08.181779 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:08.181744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:23.136090 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:23.136043    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:23.234743 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:23.234698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:23.334813 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:23.334776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:23.868803 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:23.868772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.021325 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.021290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.166600 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.166500    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"]
Jul 24 22:52:24.169786 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.169763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.172458 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.172423    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 22:52:24.172645 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.172626    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 22:52:24.172708 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.172622    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2r4tk\""
Jul 24 22:52:24.172708 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.172683    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 22:52:24.172781 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.172633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 22:52:24.189154 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.189120    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"]
Jul 24 22:52:24.265946 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.265910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-tmp\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.265946 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.265954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-manifests\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.266186 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.265972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.266186 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.265989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsgb8\" (UniqueName: \"kubernetes.io/projected/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-kube-api-access-vsgb8\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.266186 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.266083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.275266 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.275236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.366542 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.366479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-tmp\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.366719 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.366547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-manifests\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.366719 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.366572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.366719 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.366590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsgb8\" (UniqueName: \"kubernetes.io/projected/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-kube-api-access-vsgb8\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.366719 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.366624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.367006 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.366980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-manifests\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.368817 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.368795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-tmp\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.369075 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.369055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.369138 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.369057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.373089 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.373053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.375139 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.375110    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsgb8\" (UniqueName: \"kubernetes.io/projected/2f5f046f-7d2a-457e-a2f4-34b2bacdd803-kube-api-access-vsgb8\") pod \"opendatahub-operator-controller-manager-69d99768bd-l4fxb\" (UID: \"2f5f046f-7d2a-457e-a2f4-34b2bacdd803\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.474723 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.474637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.480160 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.480123    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:24.580758 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.580727    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.616376 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.616346    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"]
Jul 24 22:52:24.619066 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:52:24.619041    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5f046f_7d2a_457e_a2f4_34b2bacdd803.slice/crio-3ae70714dabedb0be0fbc8b33e21790354cb6854ec0b9edeae4250448b428841 WatchSource:0}: Error finding container 3ae70714dabedb0be0fbc8b33e21790354cb6854ec0b9edeae4250448b428841: Status 404 returned error can't find the container with id 3ae70714dabedb0be0fbc8b33e21790354cb6854ec0b9edeae4250448b428841
Jul 24 22:52:24.695027 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.694994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.796508 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.796429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.908062 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.908029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:24.991705 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:24.991675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:25.238126 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:25.238093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb" event={"ID":"2f5f046f-7d2a-457e-a2f4-34b2bacdd803","Type":"ContainerStarted","Data":"3ae70714dabedb0be0fbc8b33e21790354cb6854ec0b9edeae4250448b428841"}
Jul 24 22:52:28.251143 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:28.251102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb" event={"ID":"2f5f046f-7d2a-457e-a2f4-34b2bacdd803","Type":"ContainerStarted","Data":"7b5df7b4c90ca9d28bcfdf273bab50456aac841d612655c86ba2b7e9412b03ec"}
Jul 24 22:52:29.256113 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:29.256009    2563 generic.go:356] "Generic (PLEG): container finished" podID="2f5f046f-7d2a-457e-a2f4-34b2bacdd803" containerID="7b5df7b4c90ca9d28bcfdf273bab50456aac841d612655c86ba2b7e9412b03ec" exitCode=0
Jul 24 22:52:29.256113 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:29.256097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb" event={"ID":"2f5f046f-7d2a-457e-a2f4-34b2bacdd803","Type":"ContainerDied","Data":"7b5df7b4c90ca9d28bcfdf273bab50456aac841d612655c86ba2b7e9412b03ec"}
Jul 24 22:52:30.261969 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:30.261936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb" event={"ID":"2f5f046f-7d2a-457e-a2f4-34b2bacdd803","Type":"ContainerStarted","Data":"b4d30d783ab8a688aef86ea4299d2a90cd77034c02d49aa6441d987329c72545"}
Jul 24 22:52:30.262355 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:30.262066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:30.288800 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:30.288747    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb" podStartSLOduration=1.693525305 podStartE2EDuration="6.28872947s" totalImagesPullingTime="3.479802104s" totalInitContainerRuntime="1.115402061s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:30.286287282 +0000 UTC m=+366.919566222" watchObservedRunningTime="2026-07-24 22:52:30.28872947 +0000 UTC m=+366.922008410"
Jul 24 22:52:30.444315 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:30.444286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:30.569403 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:30.569311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:35.448850 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:35.448821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:35.545623 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:35.545591    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:35.804257 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:35.804148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:35.902635 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:35.902604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:37.494695 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:37.494653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:37.595313 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:37.595276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:41.268708 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:41.268677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-l4fxb"
Jul 24 22:52:43.211164 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:43.211128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:43.329046 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:43.329017    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:43.993176 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:43.993127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:44.056749 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:44.056697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:44.159128 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:44.159100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:44.258613 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:44.258507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:44.574135 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:44.574051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:44.825938 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:44.825850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:53.284702 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.284665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:53.334997 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.334962    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"]
Jul 24 22:52:53.338434 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.338412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:53.341319 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.341285    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vsnxp\""
Jul 24 22:52:53.341487 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.341467    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 24 22:52:53.344576 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.344551    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"]
Jul 24 22:52:53.347903 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.347881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:53.351382 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.351354    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"]
Jul 24 22:52:53.351655 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.351578    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 24 22:52:53.351787 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.351767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bpvjt\""
Jul 24 22:52:53.358576 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.358544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"]
Jul 24 22:52:53.385171 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.385126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:53.402743 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.402694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b92091da-84f6-47e5-b995-0d9090585a9f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-24r2g\" (UID: \"b92091da-84f6-47e5-b995-0d9090585a9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:53.402914 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.402751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12599807-0bc5-4b17-b082-c1d241824eda-cert\") pod \"kserve-controller-manager-d6f4b65b8-g2bfn\" (UID: \"12599807-0bc5-4b17-b082-c1d241824eda\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:53.402914 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.402778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bs5f\" (UniqueName: \"kubernetes.io/projected/12599807-0bc5-4b17-b082-c1d241824eda-kube-api-access-6bs5f\") pod \"kserve-controller-manager-d6f4b65b8-g2bfn\" (UID: \"12599807-0bc5-4b17-b082-c1d241824eda\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:53.402914 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.402894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7rk\" (UniqueName: \"kubernetes.io/projected/b92091da-84f6-47e5-b995-0d9090585a9f-kube-api-access-4l7rk\") pod \"llmisvc-controller-manager-777c6bd7d5-24r2g\" (UID: \"b92091da-84f6-47e5-b995-0d9090585a9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:53.503342 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.503308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7rk\" (UniqueName: \"kubernetes.io/projected/b92091da-84f6-47e5-b995-0d9090585a9f-kube-api-access-4l7rk\") pod \"llmisvc-controller-manager-777c6bd7d5-24r2g\" (UID: \"b92091da-84f6-47e5-b995-0d9090585a9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:53.503615 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.503362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b92091da-84f6-47e5-b995-0d9090585a9f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-24r2g\" (UID: \"b92091da-84f6-47e5-b995-0d9090585a9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:53.503615 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.503403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12599807-0bc5-4b17-b082-c1d241824eda-cert\") pod \"kserve-controller-manager-d6f4b65b8-g2bfn\" (UID: \"12599807-0bc5-4b17-b082-c1d241824eda\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:53.503615 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.503424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bs5f\" (UniqueName: \"kubernetes.io/projected/12599807-0bc5-4b17-b082-c1d241824eda-kube-api-access-6bs5f\") pod \"kserve-controller-manager-d6f4b65b8-g2bfn\" (UID: \"12599807-0bc5-4b17-b082-c1d241824eda\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:53.503615 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:52:53.503561    2563 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 24 22:52:53.503837 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:52:53.503643    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b92091da-84f6-47e5-b995-0d9090585a9f-cert podName:b92091da-84f6-47e5-b995-0d9090585a9f nodeName:}" failed. No retries permitted until 2026-07-24 22:52:54.003624952 +0000 UTC m=+390.636903870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b92091da-84f6-47e5-b995-0d9090585a9f-cert") pod "llmisvc-controller-manager-777c6bd7d5-24r2g" (UID: "b92091da-84f6-47e5-b995-0d9090585a9f") : secret "llmisvc-webhook-server-cert" not found
Jul 24 22:52:53.503837 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:52:53.503561    2563 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 24 22:52:53.503837 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:52:53.503696    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12599807-0bc5-4b17-b082-c1d241824eda-cert podName:12599807-0bc5-4b17-b082-c1d241824eda nodeName:}" failed. No retries permitted until 2026-07-24 22:52:54.00368371 +0000 UTC m=+390.636962632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/12599807-0bc5-4b17-b082-c1d241824eda-cert") pod "kserve-controller-manager-d6f4b65b8-g2bfn" (UID: "12599807-0bc5-4b17-b082-c1d241824eda") : secret "kserve-webhook-server-cert" not found
Jul 24 22:52:53.512653 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.512620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7rk\" (UniqueName: \"kubernetes.io/projected/b92091da-84f6-47e5-b995-0d9090585a9f-kube-api-access-4l7rk\") pod \"llmisvc-controller-manager-777c6bd7d5-24r2g\" (UID: \"b92091da-84f6-47e5-b995-0d9090585a9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:53.512791 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.512630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bs5f\" (UniqueName: \"kubernetes.io/projected/12599807-0bc5-4b17-b082-c1d241824eda-kube-api-access-6bs5f\") pod \"kserve-controller-manager-d6f4b65b8-g2bfn\" (UID: \"12599807-0bc5-4b17-b082-c1d241824eda\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:53.680236 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.680202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:53.775207 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:53.775171    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:54.008195 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.008112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b92091da-84f6-47e5-b995-0d9090585a9f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-24r2g\" (UID: \"b92091da-84f6-47e5-b995-0d9090585a9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:54.008195 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.008163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12599807-0bc5-4b17-b082-c1d241824eda-cert\") pod \"kserve-controller-manager-d6f4b65b8-g2bfn\" (UID: \"12599807-0bc5-4b17-b082-c1d241824eda\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:54.010682 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.010656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b92091da-84f6-47e5-b995-0d9090585a9f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-24r2g\" (UID: \"b92091da-84f6-47e5-b995-0d9090585a9f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:54.010819 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.010720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12599807-0bc5-4b17-b082-c1d241824eda-cert\") pod \"kserve-controller-manager-d6f4b65b8-g2bfn\" (UID: \"12599807-0bc5-4b17-b082-c1d241824eda\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:54.113994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.113962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:54.127743 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.127707    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"]
Jul 24 22:52:54.132954 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.132930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.136262 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.136235    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 22:52:54.136713 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.136698    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 22:52:54.137083 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.137071    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 22:52:54.137242 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.137230    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 22:52:54.148381 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.148355    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"]
Jul 24 22:52:54.208959 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.208916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfnx\" (UniqueName: \"kubernetes.io/projected/4e0160b2-3818-4707-95a8-75946cc4f81f-kube-api-access-7dfnx\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.208959 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.208964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0160b2-3818-4707-95a8-75946cc4f81f-tls-certs\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.209182 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.209034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0160b2-3818-4707-95a8-75946cc4f81f-tmp\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.253486 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.253452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:54.265630 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.265556    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:54.312217 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.310924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfnx\" (UniqueName: \"kubernetes.io/projected/4e0160b2-3818-4707-95a8-75946cc4f81f-kube-api-access-7dfnx\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.312217 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.310979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0160b2-3818-4707-95a8-75946cc4f81f-tls-certs\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.312217 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.311041    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0160b2-3818-4707-95a8-75946cc4f81f-tmp\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.314246 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.314182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0160b2-3818-4707-95a8-75946cc4f81f-tmp\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.318258 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.318167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e0160b2-3818-4707-95a8-75946cc4f81f-tls-certs\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.328674 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.328576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfnx\" (UniqueName: \"kubernetes.io/projected/4e0160b2-3818-4707-95a8-75946cc4f81f-kube-api-access-7dfnx\") pod \"kube-auth-proxy-849489fbdc-qw8vb\" (UID: \"4e0160b2-3818-4707-95a8-75946cc4f81f\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.339107 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.339050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:54.435192 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.435044    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"]
Jul 24 22:52:54.438927 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:52:54.438883    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12599807_0bc5_4b17_b082_c1d241824eda.slice/crio-4d11cde112d9a922f5d073dc60298092aae4b13afcc8d3dd5f4783bf76488add WatchSource:0}: Error finding container 4d11cde112d9a922f5d073dc60298092aae4b13afcc8d3dd5f4783bf76488add: Status 404 returned error can't find the container with id 4d11cde112d9a922f5d073dc60298092aae4b13afcc8d3dd5f4783bf76488add
Jul 24 22:52:54.442907 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.442850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"
Jul 24 22:52:54.458603 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.458581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"]
Jul 24 22:52:54.461639 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:52:54.461609    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb92091da_84f6_47e5_b995_0d9090585a9f.slice/crio-a50d90ddd51ba5070b2c693d77403b166b056ff8f8acd236980b0ed2802d614d WatchSource:0}: Error finding container a50d90ddd51ba5070b2c693d77403b166b056ff8f8acd236980b0ed2802d614d: Status 404 returned error can't find the container with id a50d90ddd51ba5070b2c693d77403b166b056ff8f8acd236980b0ed2802d614d
Jul 24 22:52:54.538134 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.538107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:54.593621 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.593579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb"]
Jul 24 22:52:54.596349 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:52:54.596309    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0160b2_3818_4707_95a8_75946cc4f81f.slice/crio-f6de23c3499a1597f6a44e61353a214b303d61d24a46534b59161a1939536cc8 WatchSource:0}: Error finding container f6de23c3499a1597f6a44e61353a214b303d61d24a46534b59161a1939536cc8: Status 404 returned error can't find the container with id f6de23c3499a1597f6a44e61353a214b303d61d24a46534b59161a1939536cc8
Jul 24 22:52:54.608251 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.608222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:54.676590 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:54.676559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:55.373051 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.373013    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb" event={"ID":"4e0160b2-3818-4707-95a8-75946cc4f81f","Type":"ContainerStarted","Data":"f6de23c3499a1597f6a44e61353a214b303d61d24a46534b59161a1939536cc8"}
Jul 24 22:52:55.375081 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.375030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn" event={"ID":"12599807-0bc5-4b17-b082-c1d241824eda","Type":"ContainerStarted","Data":"4d11cde112d9a922f5d073dc60298092aae4b13afcc8d3dd5f4783bf76488add"}
Jul 24 22:52:55.376808 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.376749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g" event={"ID":"b92091da-84f6-47e5-b995-0d9090585a9f","Type":"ContainerStarted","Data":"a50d90ddd51ba5070b2c693d77403b166b056ff8f8acd236980b0ed2802d614d"}
Jul 24 22:52:55.946342 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.946288    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"]
Jul 24 22:52:55.950084 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.950058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:55.951478 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.951450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:55.953793 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.953767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 24 22:52:55.954013 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.953991    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 24 22:52:55.954284 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.954268    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 22:52:55.954443 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.954426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 24 22:52:55.954557 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.954542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v88wh\""
Jul 24 22:52:55.970915 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:55.970879    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"]
Jul 24 22:52:56.032156 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.032116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.032351 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.032184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.032351 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.032215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v52x8\" (UniqueName: \"kubernetes.io/projected/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-kube-api-access-v52x8\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.032351 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.032245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.032351 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.032274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.032351 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.032340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.032673 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.032420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.086599 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.086563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:56.133994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.132912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.133994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.132963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.133994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.133007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.133994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.133060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.133994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.133130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.133994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.133172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.133994 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.133196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v52x8\" (UniqueName: \"kubernetes.io/projected/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-kube-api-access-v52x8\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.135497 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.135461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.139732 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.138281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.139732 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.138607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.140218 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.140166    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.141043 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.140980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.142138 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.142114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v52x8\" (UniqueName: \"kubernetes.io/projected/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-kube-api-access-v52x8\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.144087 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.144064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aef2698-81d6-4bd8-8fbb-43ca5b78af17-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p5556\" (UID: \"1aef2698-81d6-4bd8-8fbb-43ca5b78af17\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:56.267281 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:56.267184    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:52:57.163593 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:57.163564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:57.471205 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:57.471131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:52:58.267671 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.267644    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"]
Jul 24 22:52:58.270154 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:52:58.270122    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aef2698_81d6_4bd8_8fbb_43ca5b78af17.slice/crio-eba20e60bf0349d6b1a5b8003455c5d4d3ad16b67896b3b8b30b09ac7c70583d WatchSource:0}: Error finding container eba20e60bf0349d6b1a5b8003455c5d4d3ad16b67896b3b8b30b09ac7c70583d: Status 404 returned error can't find the container with id eba20e60bf0349d6b1a5b8003455c5d4d3ad16b67896b3b8b30b09ac7c70583d
Jul 24 22:52:58.393450 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.393414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb" event={"ID":"4e0160b2-3818-4707-95a8-75946cc4f81f","Type":"ContainerStarted","Data":"08a86b873b94ffa577528ea421183c1f6e00ff5095ed94ec705ae6109d0e98dd"}
Jul 24 22:52:58.395715 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.395681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn" event={"ID":"12599807-0bc5-4b17-b082-c1d241824eda","Type":"ContainerStarted","Data":"52050cb8f9c39c52891dda1fa3d99ce71ea4b9e8d3ede18323d5761d553333b9"}
Jul 24 22:52:58.395874 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.395778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:52:58.397377 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.397345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g" event={"ID":"b92091da-84f6-47e5-b995-0d9090585a9f","Type":"ContainerStarted","Data":"4a3cd58f8b91d3e0adbe1d04f889f23fd20a580135bc45e7253766d731be6e6a"}
Jul 24 22:52:58.397513 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.397407    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:52:58.398588 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.398562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556" event={"ID":"1aef2698-81d6-4bd8-8fbb-43ca5b78af17","Type":"ContainerStarted","Data":"eba20e60bf0349d6b1a5b8003455c5d4d3ad16b67896b3b8b30b09ac7c70583d"}
Jul 24 22:52:58.428556 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.428483    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-849489fbdc-qw8vb" podStartSLOduration=0.936931662 podStartE2EDuration="4.42846753s" totalImagesPullingTime="3.491535868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:58.426946005 +0000 UTC m=+395.060224946" watchObservedRunningTime="2026-07-24 22:52:58.42846753 +0000 UTC m=+395.061746469"
Jul 24 22:52:58.446974 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.446919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g" podStartSLOduration=1.813637647 podStartE2EDuration="5.446900154s" totalImagesPullingTime="3.633262507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:58.445194075 +0000 UTC m=+395.078473016" watchObservedRunningTime="2026-07-24 22:52:58.446900154 +0000 UTC m=+395.080179093"
Jul 24 22:52:58.463683 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:52:58.463629    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn" podStartSLOduration=1.816687229 podStartE2EDuration="5.463613268s" totalImagesPullingTime="3.646926039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:58.463004445 +0000 UTC m=+395.096283384" watchObservedRunningTime="2026-07-24 22:52:58.463613268 +0000 UTC m=+395.096892209"
Jul 24 22:53:00.787537 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:00.787486    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 24 22:53:00.787928 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:00.787615    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 24 22:53:01.417971 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.417933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556" event={"ID":"1aef2698-81d6-4bd8-8fbb-43ca5b78af17","Type":"ContainerStarted","Data":"0e8df9d3edb6bf9bb3399344b333a4f43687c76c74140705290d65caa5e19046"}
Jul 24 22:53:01.418168 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.418114    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:53:01.419997 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.419956    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-p5556 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 22:53:01.420133 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.420029    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556" podUID="1aef2698-81d6-4bd8-8fbb-43ca5b78af17" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 22:53:01.443759 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.443699    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556" podStartSLOduration=3.928679198 podStartE2EDuration="6.443676879s" totalImagesPullingTime="2.514997681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:53:01.442254651 +0000 UTC m=+398.075533592" watchObservedRunningTime="2026-07-24 22:53:01.443676879 +0000 UTC m=+398.076955819"
Jul 24 22:53:01.489889 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.489846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:01.589090 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.589055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:01.791591 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.791472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:01.976414 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:01.976375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:02.080385 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:02.080296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:02.424749 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:02.424715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p5556"
Jul 24 22:53:29.407505 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:29.407472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-24r2g"
Jul 24 22:53:29.411363 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:29.411341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-g2bfn"
Jul 24 22:53:43.146842 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:43.146809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:44.650414 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:44.650378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:44.749835 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:44.749804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:44.849549 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:44.849495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:45.637110 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:45.637025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:53:45.791375 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:53:45.791341    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:00.785927 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:00.785890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:00.890009 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:00.889976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:13.034614 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:13.034565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:13.622362 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:13.622330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:15.522948 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.522912    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67547fd788-5g5mm"]
Jul 24 22:54:15.525430 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.525405    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.538896 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.538864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67547fd788-5g5mm"]
Jul 24 22:54:15.584504 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.584475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:15.601722 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.601691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-config\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.601838 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.601748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-oauth-config\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.601838 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.601776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-trusted-ca-bundle\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.601838 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.601792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-serving-cert\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.601838 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.601809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-service-ca\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.602042 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.601888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdk2f\" (UniqueName: \"kubernetes.io/projected/426e6fa9-09a2-499f-9e7d-d00a18e0217e-kube-api-access-sdk2f\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.602042 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.602000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-oauth-serving-cert\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.685201 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.685168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:15.703537 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.703465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-config\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.703710 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.703577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-oauth-config\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.703710 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.703604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-trusted-ca-bundle\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.703710 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.703626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-serving-cert\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.703878 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.703786    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-service-ca\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.703878 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.703861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdk2f\" (UniqueName: \"kubernetes.io/projected/426e6fa9-09a2-499f-9e7d-d00a18e0217e-kube-api-access-sdk2f\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.703975 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.703940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-oauth-serving-cert\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.704372 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.704341    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-config\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.704556 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.704510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-service-ca\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.704629 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.704595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-oauth-serving-cert\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.704690 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.704673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/426e6fa9-09a2-499f-9e7d-d00a18e0217e-trusted-ca-bundle\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.706780 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.706757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-serving-cert\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.706892 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.706871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/426e6fa9-09a2-499f-9e7d-d00a18e0217e-console-oauth-config\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.712312 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.712278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdk2f\" (UniqueName: \"kubernetes.io/projected/426e6fa9-09a2-499f-9e7d-d00a18e0217e-kube-api-access-sdk2f\") pod \"console-67547fd788-5g5mm\" (UID: \"426e6fa9-09a2-499f-9e7d-d00a18e0217e\") " pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.785562 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.785444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:15.834856 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.834820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:15.835848 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.835825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:15.971159 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:15.971131    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67547fd788-5g5mm"]
Jul 24 22:54:15.973409 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:54:15.973382    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426e6fa9_09a2_499f_9e7d_d00a18e0217e.slice/crio-e2563baabfa25fbc9c9a1aac1ef9c84be0d87b072cf588e40798318a50418ebd WatchSource:0}: Error finding container e2563baabfa25fbc9c9a1aac1ef9c84be0d87b072cf588e40798318a50418ebd: Status 404 returned error can't find the container with id e2563baabfa25fbc9c9a1aac1ef9c84be0d87b072cf588e40798318a50418ebd
Jul 24 22:54:16.326704 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:16.326668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:16.734386 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:16.734347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67547fd788-5g5mm" event={"ID":"426e6fa9-09a2-499f-9e7d-d00a18e0217e","Type":"ContainerStarted","Data":"ab404d29f01dffdee351922c474b353dcdde57a7f0ead638a6857c25f959797f"}
Jul 24 22:54:16.734803 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:16.734406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67547fd788-5g5mm" event={"ID":"426e6fa9-09a2-499f-9e7d-d00a18e0217e","Type":"ContainerStarted","Data":"e2563baabfa25fbc9c9a1aac1ef9c84be0d87b072cf588e40798318a50418ebd"}
Jul 24 22:54:16.752718 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:16.752659    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67547fd788-5g5mm" podStartSLOduration=1.752639736 podStartE2EDuration="1.752639736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:54:16.751411246 +0000 UTC m=+473.384690187" watchObservedRunningTime="2026-07-24 22:54:16.752639736 +0000 UTC m=+473.385918675"
Jul 24 22:54:18.430830 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:18.430796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:20.292262 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.292224    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"]
Jul 24 22:54:20.295499 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.295468    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:20.299031 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.299003    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 22:54:20.299762 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.299730    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bqwlf\""
Jul 24 22:54:20.299888 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.299764    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 22:54:20.309285 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.309256    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"]
Jul 24 22:54:20.349933 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.349889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zllgs\" (UniqueName: \"kubernetes.io/projected/1b7ebc94-6821-4e4f-b172-a138d3e4920a-kube-api-access-zllgs\") pod \"limitador-operator-controller-manager-859d7b78bd-7rz9v\" (UID: \"1b7ebc94-6821-4e4f-b172-a138d3e4920a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:20.450690 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.450647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zllgs\" (UniqueName: \"kubernetes.io/projected/1b7ebc94-6821-4e4f-b172-a138d3e4920a-kube-api-access-zllgs\") pod \"limitador-operator-controller-manager-859d7b78bd-7rz9v\" (UID: \"1b7ebc94-6821-4e4f-b172-a138d3e4920a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:20.465212 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.465176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zllgs\" (UniqueName: \"kubernetes.io/projected/1b7ebc94-6821-4e4f-b172-a138d3e4920a-kube-api-access-zllgs\") pod \"limitador-operator-controller-manager-859d7b78bd-7rz9v\" (UID: \"1b7ebc94-6821-4e4f-b172-a138d3e4920a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:20.477990 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.477960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:20.606741 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.606706    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:20.758736 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:20.758708    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"]
Jul 24 22:54:20.760846 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:54:20.760814    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7ebc94_6821_4e4f_b172_a138d3e4920a.slice/crio-672b4b8fe562a85e009f488744ad6af7b9973d70c38b83c2ace5af26eedec1f5 WatchSource:0}: Error finding container 672b4b8fe562a85e009f488744ad6af7b9973d70c38b83c2ace5af26eedec1f5: Status 404 returned error can't find the container with id 672b4b8fe562a85e009f488744ad6af7b9973d70c38b83c2ace5af26eedec1f5
Jul 24 22:54:21.449761 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:21.449713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:21.759276 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:21.759179    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v" event={"ID":"1b7ebc94-6821-4e4f-b172-a138d3e4920a","Type":"ContainerStarted","Data":"672b4b8fe562a85e009f488744ad6af7b9973d70c38b83c2ace5af26eedec1f5"}
Jul 24 22:54:22.282094 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:22.282062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:22.765653 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:22.765612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v" event={"ID":"1b7ebc94-6821-4e4f-b172-a138d3e4920a","Type":"ContainerStarted","Data":"5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5"}
Jul 24 22:54:22.766090 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:22.765744    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:22.786090 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:22.786034    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v" podStartSLOduration=1.101206933 podStartE2EDuration="2.786014643s" totalImagesPullingTime="1.68480771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:54:22.784771157 +0000 UTC m=+479.418050097" watchObservedRunningTime="2026-07-24 22:54:22.786014643 +0000 UTC m=+479.419293583"
Jul 24 22:54:24.465825 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:24.465779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:24.902600 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:24.902565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:25.835906 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:25.835859    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:25.835906 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:25.835922    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:25.841939 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:25.841912    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:26.788366 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:26.788339    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67547fd788-5g5mm"
Jul 24 22:54:26.844465 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:26.844422    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd5b5b74b-v75cs"]
Jul 24 22:54:26.963822 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:26.963792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:27.064597 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:27.064495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:27.166605 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:27.166569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:28.111489 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:28.111446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:30.042156 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:30.042127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:30.140726 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:30.140689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:30.452290 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:30.452255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:30.554075 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:30.554037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:33.772992 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:33.772954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:33.839710 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:33.839663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:33.940387 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:33.940352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:36.689769 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:36.689730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:37.041823 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:37.041740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:37.249170 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:37.249139    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:37.308289 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:37.308200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:41.198588 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:41.198548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:41.297794 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:41.297761    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:46.217833 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.217790    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"]
Jul 24 22:54:46.221245 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.221225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:46.224059 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.224034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l9czn\""
Jul 24 22:54:46.241157 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.241126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"]
Jul 24 22:54:46.382512 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.382476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdvl\" (UniqueName: \"kubernetes.io/projected/b11c9674-b988-4475-a90c-cf2a1522f0f8-kube-api-access-mfdvl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" (UID: \"b11c9674-b988-4475-a90c-cf2a1522f0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:46.382726 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.382614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b11c9674-b988-4475-a90c-cf2a1522f0f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" (UID: \"b11c9674-b988-4475-a90c-cf2a1522f0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:46.483964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.483862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdvl\" (UniqueName: \"kubernetes.io/projected/b11c9674-b988-4475-a90c-cf2a1522f0f8-kube-api-access-mfdvl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" (UID: \"b11c9674-b988-4475-a90c-cf2a1522f0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:46.483964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.483932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b11c9674-b988-4475-a90c-cf2a1522f0f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" (UID: \"b11c9674-b988-4475-a90c-cf2a1522f0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:46.484278 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.484259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b11c9674-b988-4475-a90c-cf2a1522f0f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" (UID: \"b11c9674-b988-4475-a90c-cf2a1522f0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:46.495419 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.495385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdvl\" (UniqueName: \"kubernetes.io/projected/b11c9674-b988-4475-a90c-cf2a1522f0f8-kube-api-access-mfdvl\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" (UID: \"b11c9674-b988-4475-a90c-cf2a1522f0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:46.532275 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.532236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:46.675591 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.675563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"]
Jul 24 22:54:46.677804 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:54:46.677772    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11c9674_b988_4475_a90c_cf2a1522f0f8.slice/crio-79df86e54a0db345f0dd12126fa0a8986ef8625dff89e37e8a50ce12cbba0579 WatchSource:0}: Error finding container 79df86e54a0db345f0dd12126fa0a8986ef8625dff89e37e8a50ce12cbba0579: Status 404 returned error can't find the container with id 79df86e54a0db345f0dd12126fa0a8986ef8625dff89e37e8a50ce12cbba0579
Jul 24 22:54:46.874069 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.874031    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5" event={"ID":"b11c9674-b988-4475-a90c-cf2a1522f0f8","Type":"ContainerStarted","Data":"79df86e54a0db345f0dd12126fa0a8986ef8625dff89e37e8a50ce12cbba0579"}
Jul 24 22:54:46.944935 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.944897    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"]
Jul 24 22:54:46.950926 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.950894    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"]
Jul 24 22:54:46.994412 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.994371    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"]
Jul 24 22:54:46.994632 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:46.994601    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v" podUID="1b7ebc94-6821-4e4f-b172-a138d3e4920a" containerName="manager" containerID="cri-o://5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5" gracePeriod=2
Jul 24 22:54:47.017957 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.017918    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"]
Jul 24 22:54:47.024458 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:47.024407    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1b7ebc94-6821-4e4f-b172-a138d3e4920a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-7rz9v\": pods \"limitador-operator-controller-manager-859d7b78bd-7rz9v\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:47.026663 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.026631    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"]
Jul 24 22:54:47.026955 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.026943    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b7ebc94-6821-4e4f-b172-a138d3e4920a" containerName="manager"
Jul 24 22:54:47.026996 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.026956    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b7ebc94-6821-4e4f-b172-a138d3e4920a" containerName="manager"
Jul 24 22:54:47.027030 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.027023    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b7ebc94-6821-4e4f-b172-a138d3e4920a" containerName="manager"
Jul 24 22:54:47.030022 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.029999    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"
Jul 24 22:54:47.032397 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:47.032362    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7rz9v\" is forbidden: User \"system:node:ip-10-0-136-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-221.ec2.internal' and this object" podUID="1b7ebc94-6821-4e4f-b172-a138d3e4920a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:47.041058 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.041013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"]
Jul 24 22:54:47.087468 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.087413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfgwh\" (UniqueName: \"kubernetes.io/projected/3752acfb-9605-4c60-8f05-262b01003338-kube-api-access-wfgwh\") pod \"limitador-operator-controller-manager-859d7b78bd-kzwnr\" (UID: \"3752acfb-9605-4c60-8f05-262b01003338\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"
Jul 24 22:54:47.188228 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.188192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfgwh\" (UniqueName: \"kubernetes.io/projected/3752acfb-9605-4c60-8f05-262b01003338-kube-api-access-wfgwh\") pod \"limitador-operator-controller-manager-859d7b78bd-kzwnr\" (UID: \"3752acfb-9605-4c60-8f05-262b01003338\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"
Jul 24 22:54:47.201306 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.201275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfgwh\" (UniqueName: \"kubernetes.io/projected/3752acfb-9605-4c60-8f05-262b01003338-kube-api-access-wfgwh\") pod \"limitador-operator-controller-manager-859d7b78bd-kzwnr\" (UID: \"3752acfb-9605-4c60-8f05-262b01003338\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"
Jul 24 22:54:47.323736 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.323709    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:47.326410 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:47.326372    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7rz9v\" is forbidden: User \"system:node:ip-10-0-136-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-221.ec2.internal' and this object" podUID="1b7ebc94-6821-4e4f-b172-a138d3e4920a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:47.364292 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.364246    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"
Jul 24 22:54:47.390477 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.390447    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zllgs\" (UniqueName: \"kubernetes.io/projected/1b7ebc94-6821-4e4f-b172-a138d3e4920a-kube-api-access-zllgs\") pod \"1b7ebc94-6821-4e4f-b172-a138d3e4920a\" (UID: \"1b7ebc94-6821-4e4f-b172-a138d3e4920a\") "
Jul 24 22:54:47.393200 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.393157    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7ebc94-6821-4e4f-b172-a138d3e4920a-kube-api-access-zllgs" (OuterVolumeSpecName: "kube-api-access-zllgs") pod "1b7ebc94-6821-4e4f-b172-a138d3e4920a" (UID: "1b7ebc94-6821-4e4f-b172-a138d3e4920a"). InnerVolumeSpecName "kube-api-access-zllgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:54:47.491154 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.491072    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zllgs\" (UniqueName: \"kubernetes.io/projected/1b7ebc94-6821-4e4f-b172-a138d3e4920a-kube-api-access-zllgs\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:47.732969 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.732940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"]
Jul 24 22:54:47.735700 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:54:47.735670    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3752acfb_9605_4c60_8f05_262b01003338.slice/crio-aabe48eb4a466b7698bc4d35abaa69f2927cdce33d7fc338d6116b57849e273f WatchSource:0}: Error finding container aabe48eb4a466b7698bc4d35abaa69f2927cdce33d7fc338d6116b57849e273f: Status 404 returned error can't find the container with id aabe48eb4a466b7698bc4d35abaa69f2927cdce33d7fc338d6116b57849e273f
Jul 24 22:54:47.841954 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.841924    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b7ebc94-6821-4e4f-b172-a138d3e4920a" path="/var/lib/kubelet/pods/1b7ebc94-6821-4e4f-b172-a138d3e4920a/volumes"
Jul 24 22:54:47.880685 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.880652    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b7ebc94-6821-4e4f-b172-a138d3e4920a" containerID="5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5" exitCode=0
Jul 24 22:54:47.880870 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.880709    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7rz9v"
Jul 24 22:54:47.880870 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.880769    2563 scope.go:117] "RemoveContainer" containerID="5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5"
Jul 24 22:54:47.883834 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.883787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr" event={"ID":"3752acfb-9605-4c60-8f05-262b01003338","Type":"ContainerStarted","Data":"4cac03838c25b94f75f0430b497bca170d1e7b9dabe59159b06917ae20c69ead"}
Jul 24 22:54:47.883958 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.883855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr" event={"ID":"3752acfb-9605-4c60-8f05-262b01003338","Type":"ContainerStarted","Data":"aabe48eb4a466b7698bc4d35abaa69f2927cdce33d7fc338d6116b57849e273f"}
Jul 24 22:54:47.911055 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.910988    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr" podStartSLOduration=1.9109650249999999 podStartE2EDuration="1.910965025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:54:47.909848589 +0000 UTC m=+504.543127529" watchObservedRunningTime="2026-07-24 22:54:47.910965025 +0000 UTC m=+504.544243966"
Jul 24 22:54:47.915422 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.915394    2563 scope.go:117] "RemoveContainer" containerID="5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5"
Jul 24 22:54:47.915884 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:47.915854    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5\": container with ID starting with 5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5 not found: ID does not exist" containerID="5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5"
Jul 24 22:54:47.915977 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:47.915900    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5"} err="failed to get container status \"5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5\": rpc error: code = NotFound desc = could not find container \"5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5\": container with ID starting with 5de545a02e54c33df1c05c32698d0bd521135dc1794c315d20084e2badfb65a5 not found: ID does not exist"
Jul 24 22:54:48.891362 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:48.891326    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"
Jul 24 22:54:49.038539 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:49.038486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:49.144016 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:49.143930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:49.493555 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:49.493454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:49.593284 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:49.593245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:51.864086 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:51.864052    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fd5b5b74b-v75cs" podUID="ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" containerName="console" containerID="cri-o://abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81" gracePeriod=15
Jul 24 22:54:51.908568 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:51.908490    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" containerName="manager" containerID="cri-o://bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b" gracePeriod=2
Jul 24 22:54:52.157895 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.157862    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:54:52.179049 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.179015    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:52.181241 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:52.181203    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" is forbidden: User \"system:node:ip-10-0-136-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-221.ec2.internal' and this object" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:52.236150 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236113    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-oauth-serving-cert\") pod \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") "
Jul 24 22:54:52.236150 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236151    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-serving-cert\") pod \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") "
Jul 24 22:54:52.236394 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236176    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfdvl\" (UniqueName: \"kubernetes.io/projected/b11c9674-b988-4475-a90c-cf2a1522f0f8-kube-api-access-mfdvl\") pod \"b11c9674-b988-4475-a90c-cf2a1522f0f8\" (UID: \"b11c9674-b988-4475-a90c-cf2a1522f0f8\") "
Jul 24 22:54:52.236394 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236195    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-service-ca\") pod \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") "
Jul 24 22:54:52.236394 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-oauth-config\") pod \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") "
Jul 24 22:54:52.236394 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236260    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b11c9674-b988-4475-a90c-cf2a1522f0f8-extensions-socket-volume\") pod \"b11c9674-b988-4475-a90c-cf2a1522f0f8\" (UID: \"b11c9674-b988-4475-a90c-cf2a1522f0f8\") "
Jul 24 22:54:52.236394 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236288    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-trusted-ca-bundle\") pod \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") "
Jul 24 22:54:52.236394 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-config\") pod \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") "
Jul 24 22:54:52.236394 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mrb4\" (UniqueName: \"kubernetes.io/projected/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-kube-api-access-7mrb4\") pod \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\" (UID: \"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab\") "
Jul 24 22:54:52.236783 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b11c9674-b988-4475-a90c-cf2a1522f0f8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b11c9674-b988-4475-a90c-cf2a1522f0f8" (UID: "b11c9674-b988-4475-a90c-cf2a1522f0f8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 22:54:52.236783 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236620    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" (UID: "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:54:52.236783 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-service-ca" (OuterVolumeSpecName: "service-ca") pod "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" (UID: "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:54:52.236783 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236719    2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-oauth-serving-cert\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.236783 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236737    2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.236783 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236751    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b11c9674-b988-4475-a90c-cf2a1522f0f8-extensions-socket-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.237075 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.236974    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-config" (OuterVolumeSpecName: "console-config") pod "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" (UID: "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:54:52.237135 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.237081    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" (UID: "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 22:54:52.238511 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.238486    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" (UID: "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:54:52.238511 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.238505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" (UID: "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 22:54:52.238815 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.238796    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-kube-api-access-7mrb4" (OuterVolumeSpecName: "kube-api-access-7mrb4") pod "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" (UID: "ec1eb538-ce1c-42e4-9ebe-be610a91e1ab"). InnerVolumeSpecName "kube-api-access-7mrb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:54:52.238871 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.238830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11c9674-b988-4475-a90c-cf2a1522f0f8-kube-api-access-mfdvl" (OuterVolumeSpecName: "kube-api-access-mfdvl") pod "b11c9674-b988-4475-a90c-cf2a1522f0f8" (UID: "b11c9674-b988-4475-a90c-cf2a1522f0f8"). InnerVolumeSpecName "kube-api-access-mfdvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 22:54:52.337673 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.337627    2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-serving-cert\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.337673 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.337664    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfdvl\" (UniqueName: \"kubernetes.io/projected/b11c9674-b988-4475-a90c-cf2a1522f0f8-kube-api-access-mfdvl\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.337673 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.337677    2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-oauth-config\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.337910 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.337688    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-trusted-ca-bundle\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.337910 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.337700    2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-console-config\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.337910 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.337712    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrb4\" (UniqueName: \"kubernetes.io/projected/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab-kube-api-access-7mrb4\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 22:54:52.914226 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.914187    2563 generic.go:356] "Generic (PLEG): container finished" podID="b11c9674-b988-4475-a90c-cf2a1522f0f8" containerID="bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b" exitCode=2
Jul 24 22:54:52.914715 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.914234    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:52.914715 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.914292    2563 scope.go:117] "RemoveContainer" containerID="bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b"
Jul 24 22:54:52.915761 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.915737    2563 generic.go:356] "Generic (PLEG): container finished" podID="ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" containerID="abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81" exitCode=0
Jul 24 22:54:52.915823 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.915787    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fd5b5b74b-v75cs"
Jul 24 22:54:52.915880 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.915813    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd5b5b74b-v75cs" event={"ID":"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab","Type":"ContainerDied","Data":"abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81"}
Jul 24 22:54:52.915880 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.915867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fd5b5b74b-v75cs" event={"ID":"ec1eb538-ce1c-42e4-9ebe-be610a91e1ab","Type":"ContainerDied","Data":"a922901218a6398f1c81cf292914d902e2db98fccd7edd9903a2f90fddbed365"}
Jul 24 22:54:52.916896 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:52.916867    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" is forbidden: User \"system:node:ip-10-0-136-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-221.ec2.internal' and this object" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:52.919167 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:52.919139    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" is forbidden: User \"system:node:ip-10-0-136-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-221.ec2.internal' and this object" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:52.923590 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.923567    2563 scope.go:117] "RemoveContainer" containerID="bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b"
Jul 24 22:54:52.923925 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:52.923907    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b\": container with ID starting with bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b not found: ID does not exist" containerID="bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b"
Jul 24 22:54:52.923997 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.923935    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b"} err="failed to get container status \"bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b\": rpc error: code = NotFound desc = could not find container \"bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b\": container with ID starting with bc15f3a7a0c93a451f1f1050be6f4bea957774701804ad3cf4e408474e604b7b not found: ID does not exist"
Jul 24 22:54:52.923997 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.923953    2563 scope.go:117] "RemoveContainer" containerID="abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81"
Jul 24 22:54:52.928050 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:52.928021    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" is forbidden: User \"system:node:ip-10-0-136-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-221.ec2.internal' and this object" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:52.932576 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.932559    2563 scope.go:117] "RemoveContainer" containerID="abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81"
Jul 24 22:54:52.932900 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:52.932879    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81\": container with ID starting with abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81 not found: ID does not exist" containerID="abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81"
Jul 24 22:54:52.932960 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.932913    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81"} err="failed to get container status \"abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81\": rpc error: code = NotFound desc = could not find container \"abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81\": container with ID starting with abbbd6446373a2058af954b338014f0ae59a1428f3cbaeed05279cccec789d81 not found: ID does not exist"
Jul 24 22:54:52.944627 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.944592    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fd5b5b74b-v75cs"]
Jul 24 22:54:52.950955 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:52.950918    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" is forbidden: User \"system:node:ip-10-0-136-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-221.ec2.internal' and this object" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:52.951123 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:52.951091    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fd5b5b74b-v75cs"]
Jul 24 22:54:53.025249 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:53.025218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:53.125475 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:53.125436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:53.843063 ip-10-0-136-221 kubenswrapper[2563]: E0724 22:54:53.843022    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gkgc5\" is forbidden: User \"system:node:ip-10-0-136-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-221.ec2.internal' and this object" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gkgc5"
Jul 24 22:54:53.845487 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:53.845457    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" path="/var/lib/kubelet/pods/b11c9674-b988-4475-a90c-cf2a1522f0f8/volumes"
Jul 24 22:54:53.845856 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:53.845843    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" path="/var/lib/kubelet/pods/ec1eb538-ce1c-42e4-9ebe-be610a91e1ab/volumes"
Jul 24 22:54:55.626187 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:55.626151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:55.727372 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:55.727339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:54:59.899827 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:54:59.899796    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kzwnr"
Jul 24 22:55:00.303233 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:00.303146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:00.403706 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:00.403673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:32.329435 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:32.329399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:32.426990 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:32.426959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:32.608374 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:32.608328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:32.743420 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:32.743370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:34.546165 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.546132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:34.654118 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.654074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:34.710992 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.710966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:34.935419 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.935381    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:55:34.935716 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.935703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" containerName="console"
Jul 24 22:55:34.935768 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.935718    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" containerName="console"
Jul 24 22:55:34.935768 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.935734    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" containerName="manager"
Jul 24 22:55:34.935768 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.935740    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" containerName="manager"
Jul 24 22:55:34.935858 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.935797    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11c9674-b988-4475-a90c-cf2a1522f0f8" containerName="manager"
Jul 24 22:55:34.935858 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.935806    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1eb538-ce1c-42e4-9ebe-be610a91e1ab" containerName="console"
Jul 24 22:55:34.938974 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.938948    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:34.941548 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.941508    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 22:55:34.941799 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.941631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pqjc8\""
Jul 24 22:55:34.949580 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.949554    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:55:34.980706 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:34.980665    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:55:35.103412 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.103374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35-config-file\") pod \"limitador-limitador-54d6d988d5-dmgs6\" (UID: \"af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:35.103682 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.103431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wdw\" (UniqueName: \"kubernetes.io/projected/af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35-kube-api-access-m7wdw\") pod \"limitador-limitador-54d6d988d5-dmgs6\" (UID: \"af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:35.204713 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.204621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35-config-file\") pod \"limitador-limitador-54d6d988d5-dmgs6\" (UID: \"af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:35.204713 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.204701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wdw\" (UniqueName: \"kubernetes.io/projected/af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35-kube-api-access-m7wdw\") pod \"limitador-limitador-54d6d988d5-dmgs6\" (UID: \"af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:35.205353 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.205314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35-config-file\") pod \"limitador-limitador-54d6d988d5-dmgs6\" (UID: \"af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:35.213768 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.213728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wdw\" (UniqueName: \"kubernetes.io/projected/af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35-kube-api-access-m7wdw\") pod \"limitador-limitador-54d6d988d5-dmgs6\" (UID: \"af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:35.249624 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.249581    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:35.399453 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.399415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:55:35.402928 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.402895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:35.461102 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:35.460987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:36.119371 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:36.119316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6" event={"ID":"af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35","Type":"ContainerStarted","Data":"427ca4c142ad25e49d7cf590a07325e5610d269f7edcbd3155a7850d803f8039"}
Jul 24 22:55:38.130628 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:38.130504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6" event={"ID":"af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35","Type":"ContainerStarted","Data":"6537fa43379ed9c36f8ce0bd2e2bd5bcc46604de4bedd1f87c2d216051fa2609"}
Jul 24 22:55:38.131076 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:38.130638    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:38.149594 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:38.149540    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6" podStartSLOduration=1.6861759589999998 podStartE2EDuration="4.149504917s" totalImagesPullingTime="2.463328958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:55:38.147759659 +0000 UTC m=+554.781038601" watchObservedRunningTime="2026-07-24 22:55:38.149504917 +0000 UTC m=+554.782783856"
Jul 24 22:55:49.135607 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:49.135571    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"
Jul 24 22:55:50.667397 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:50.667366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:50.767039 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:50.767003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:55:51.134239 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:55:51.134206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:02.756367 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:02.756328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:02.923913 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:02.923842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:09.365721 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:09.365683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:09.461727 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:09.461696    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:09.516703 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:09.516656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:09.586168 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:09.586137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:12.539930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:12.539892    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:12.643662 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:12.643629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:23.739879 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:23.739850    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 22:56:23.741376 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:23.741356    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 22:56:27.796163 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:27.796129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:40.348942 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:40.348906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:40.444458 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:40.444425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:41.097930 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:41.097894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:41.276971 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:41.276944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:41.652940 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:41.652905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:41.752949 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:41.752923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:41.843008 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:41.842975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:56:50.456662 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:50.456626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:50.505566 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:50.505507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:50.667988 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:50.667953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:56:50.769258 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:56:50.769175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:57:18.676418 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:18.676389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:57:19.228868 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:19.228829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:57:40.776574 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:40.776539    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:57:41.522856 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:41.522816    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:57:42.837324 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.837288    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"]
Jul 24 22:57:42.840024 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.840008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:42.843934 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.843906    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 22:57:42.844073 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.843906    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4rw57\""
Jul 24 22:57:42.844073 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.844029    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 24 22:57:42.844208 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.844145    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 22:57:42.848687 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.848655    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"]
Jul 24 22:57:42.899464 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.899419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffqc\" (UniqueName: \"kubernetes.io/projected/28ef1232-1949-4b38-96d4-0a38ceb9b805-kube-api-access-pffqc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:42.899464 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.899466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:42.899693 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.899570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:42.899693 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.899596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:42.899693 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.899635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ef1232-1949-4b38-96d4-0a38ceb9b805-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:42.899792 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:42.899704    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.000777 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.000742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffqc\" (UniqueName: \"kubernetes.io/projected/28ef1232-1949-4b38-96d4-0a38ceb9b805-kube-api-access-pffqc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.000964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.000785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.000964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.000822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.000964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.000845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.000964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.000900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ef1232-1949-4b38-96d4-0a38ceb9b805-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.000964 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.000943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.001273 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.001250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.001273 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.001267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.001597 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.001568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.003200 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.003167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ef1232-1949-4b38-96d4-0a38ceb9b805-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.003573 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.003545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ef1232-1949-4b38-96d4-0a38ceb9b805-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.008720 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.008695    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffqc\" (UniqueName: \"kubernetes.io/projected/28ef1232-1949-4b38-96d4-0a38ceb9b805-kube-api-access-pffqc\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tkdz4\" (UID: \"28ef1232-1949-4b38-96d4-0a38ceb9b805\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.150710 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.150679    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:43.160862 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.160837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:57:43.283883 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.283859    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"]
Jul 24 22:57:43.286570 ip-10-0-136-221 kubenswrapper[2563]: W0724 22:57:43.286539    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ef1232_1949_4b38_96d4_0a38ceb9b805.slice/crio-768601f4ae853a3e23f79982e8b74ee1eade6d24f3316dc0e339da37d6560944 WatchSource:0}: Error finding container 768601f4ae853a3e23f79982e8b74ee1eade6d24f3316dc0e339da37d6560944: Status 404 returned error can't find the container with id 768601f4ae853a3e23f79982e8b74ee1eade6d24f3316dc0e339da37d6560944
Jul 24 22:57:43.287952 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.287937    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 22:57:43.684113 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:43.684073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4" event={"ID":"28ef1232-1949-4b38-96d4-0a38ceb9b805","Type":"ContainerStarted","Data":"768601f4ae853a3e23f79982e8b74ee1eade6d24f3316dc0e339da37d6560944"}
Jul 24 22:57:44.690377 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:44.690341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4" event={"ID":"28ef1232-1949-4b38-96d4-0a38ceb9b805","Type":"ContainerStarted","Data":"47de74779164f2ae26f37f0776cd52f03a5505233b5c7d244697b4023d1fe5a2"}
Jul 24 22:57:44.709234 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:44.709180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4" podStartSLOduration=1.776072703 podStartE2EDuration="2.709158876s" totalImagesPullingTime="933.086173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:57:44.708445984 +0000 UTC m=+681.341724928" watchObservedRunningTime="2026-07-24 22:57:44.709158876 +0000 UTC m=+681.342437815"
Jul 24 22:57:47.436554 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:47.435401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:57:53.151684 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:53.151643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:53.152174 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:53.151714    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:53.164320 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:53.164293    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:53.743357 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:53.743327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tkdz4"
Jul 24 22:57:59.190768 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:57:59.190728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:58:00.535910 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:00.535874    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:58:02.464393 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:02.464360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:58:02.513750 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:02.513709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:58:11.555098 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:11.555065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:58:11.656175 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:11.656141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:58:17.286944 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:17.286911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:58:18.603086 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:18.603052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:58:20.036444 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:20.036404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:58:30.570903 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:30.570865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:58:33.028973 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:33.028935    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 22:58:49.302962 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:58:49.302926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:59:37.402215 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:59:37.402174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:59:37.504728 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:59:37.504695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:59:38.549192 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:59:38.549158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 22:59:38.647079 ip-10-0-136-221 kubenswrapper[2563]: I0724 22:59:38.647042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:00:00.145420 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:00.145342    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k"]
Jul 24 23:00:00.148665 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:00.148645    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k"
Jul 24 23:00:00.151305 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:00.151280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:00:00.160661 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:00.160623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k"
Jul 24 23:00:00.165975 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:00.165736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k"]
Jul 24 23:00:00.299154 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:00.299124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k"]
Jul 24 23:00:00.301835 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:00:00.301806    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397f99d4_68ac_40a5_a639_dd36ed36ac4a.slice/crio-32c5b8c736957b0002403118db7ae4052b853ec419f8c3369159d9e2bb39babd WatchSource:0}: Error finding container 32c5b8c736957b0002403118db7ae4052b853ec419f8c3369159d9e2bb39babd: Status 404 returned error can't find the container with id 32c5b8c736957b0002403118db7ae4052b853ec419f8c3369159d9e2bb39babd
Jul 24 23:00:01.295653 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:01.295616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k" event={"ID":"397f99d4-68ac-40a5-a639-dd36ed36ac4a","Type":"ContainerStarted","Data":"32c5b8c736957b0002403118db7ae4052b853ec419f8c3369159d9e2bb39babd"}
Jul 24 23:00:04.311607 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:04.311489    2563 generic.go:356] "Generic (PLEG): container finished" podID="397f99d4-68ac-40a5-a639-dd36ed36ac4a" containerID="5f36b3ba96341510c402aa8fae7d8c63387af77e37f3cd291b3d86ce7e1dbc28" exitCode=0
Jul 24 23:00:04.311607 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:04.311577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k" event={"ID":"397f99d4-68ac-40a5-a639-dd36ed36ac4a","Type":"ContainerDied","Data":"5f36b3ba96341510c402aa8fae7d8c63387af77e37f3cd291b3d86ce7e1dbc28"}
Jul 24 23:00:05.453864 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:05.453833    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k"
Jul 24 23:00:06.321704 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:06.321616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k" event={"ID":"397f99d4-68ac-40a5-a639-dd36ed36ac4a","Type":"ContainerDied","Data":"32c5b8c736957b0002403118db7ae4052b853ec419f8c3369159d9e2bb39babd"}
Jul 24 23:00:06.321704 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:06.321643    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748900-7nm6k"
Jul 24 23:00:06.321704 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:06.321669    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c5b8c736957b0002403118db7ae4052b853ec419f8c3369159d9e2bb39babd"
Jul 24 23:00:07.337235 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:07.337197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:00:11.970055 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:11.970022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:00:12.026739 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:12.026699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:00:17.037257 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:17.037218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:00:27.734766 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:00:27.734727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:01:23.773818 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:01:23.773784    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:01:23.774286 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:01:23.773837    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:01:42.597404 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:01:42.597364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:01:45.350631 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:01:45.350596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:10.733174 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:10.733125    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:02:22.523660 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:22.523617    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:02:29.228424 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:29.228389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:31.545651 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:31.545602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:31.644146 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:31.644115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:31.764122 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:31.764089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:37.590747 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:37.590712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:37.652386 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:37.652355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:37.733953 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:37.733918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:37.833759 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:37.833723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:37.980919 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:37.980886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:43.505017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:43.504977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:02:43.604886 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:02:43.604838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:03:01.930718 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:03:01.930628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:03:10.322894 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:03:10.322854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:03:20.925233 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:03:20.925192    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:03:29.132388 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:03:29.132345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:03:40.441593 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:03:40.441551    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:03:49.059938 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:03:49.059897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:03:50.129008 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:03:50.128970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:04:01.023693 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:04:01.023656    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:04:10.927144 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:04:10.927104    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:04:33.528767 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:04:33.528733    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:04:43.728576 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:04:43.728537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:05:08.332190 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:05:08.332151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:05:18.833877 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:05:18.833837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:05:22.911432 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:05:22.911387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:05:23.000200 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:05:23.000163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:05:34.428972 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:05:34.428932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:05:43.842586 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:05:43.842548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:05:53.346840 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:05:53.346737    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:06:02.732628 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:06:02.732578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:06:11.234819 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:06:11.234781    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:06:23.803261 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:06:23.803234    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:06:23.803884 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:06:23.803726    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:07:33.717630 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:07:33.717593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:08:00.828576 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:08:00.828537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:08:10.129511 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:08:10.129472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:08:17.019587 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:08:17.019508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:08:17.081772 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:08:17.081741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:08:22.628869 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:08:22.628828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:08:35.737357 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:08:35.737317    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:08:46.221576 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:08:46.221478    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:08:55.229576 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:08:55.229538    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:09:04.741313 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:09:04.741266    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:09:14.338210 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:09:14.338172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:09:25.239898 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:09:25.239861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:09:35.429625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:09:35.429588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:09:48.130759 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:09:48.130722    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:10:09.535973 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:09.535926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:10:18.151026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:18.150991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:10:18.355399 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:18.355357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:10:19.236090 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:19.236052    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:10:29.440234 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:29.440195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:10:31.325844 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:31.325802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:10:36.296695 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:36.296641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:10:39.338416 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:39.338377    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:10:41.480617 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:10:41.480584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:11:06.796362 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:06.796325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:11:08.560094 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:08.560056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:11:09.638879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:09.638843    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:11:23.831233 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:23.831195    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:11:23.831973 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:23.831957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:11:25.030392 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:25.030354    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:11:31.828333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:31.828303    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:11:41.331373 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:41.331331    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:11:49.036964 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:49.036872    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:11:58.632850 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:11:58.632803    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:12:07.822991 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:12:07.822949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:12:18.075531 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:12:18.075466    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:12:26.931654 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:12:26.931605    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:12:36.224910 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:12:36.224871    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:12:49.528106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:12:49.528066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:12:58.727746 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:12:58.727708    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:13:08.735101 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:13:08.735061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:13:21.138893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:13:21.138851    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:13:29.832032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:13:29.831991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:13:36.931591 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:13:36.931551    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:13:46.525549 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:13:46.525486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:13:54.332484 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:13:54.332445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:14:06.328229 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:06.328186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:14:18.530657 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:18.530620    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:14:21.423315 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:21.423283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:28.326171 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:28.326135    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:14:47.030433 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:47.030350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:50.032923 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:50.032889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:50.133150 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:50.133119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:50.488476 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:50.488444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:50.529382 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:50.529344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:14:56.282193 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:56.282160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:56.382387 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:56.382354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:56.506574 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:56.506538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:56.820507 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:56.820478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:57.177733 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:57.177698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:59.213386 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.213352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:59.294490 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.294459    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"]
Jul 24 23:14:59.294865 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.294849    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="397f99d4-68ac-40a5-a639-dd36ed36ac4a" containerName="cleanup"
Jul 24 23:14:59.294947 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.294869    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="397f99d4-68ac-40a5-a639-dd36ed36ac4a" containerName="cleanup"
Jul 24 23:14:59.295005 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.294992    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="397f99d4-68ac-40a5-a639-dd36ed36ac4a" containerName="cleanup"
Jul 24 23:14:59.297809 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.297786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.298230 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.298196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:59.300644 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.300614    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:14:59.300765 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.300680    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:14:59.301797 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.301772    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e390b4a-serving-cert\""
Jul 24 23:14:59.301901 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.301801    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:14:59.306964 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.306937    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"]
Jul 24 23:14:59.325135 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.325096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.325321 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.325161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e26084-1f0d-4929-b88c-485fd03652d6-openshift-service-ca\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.325321 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.325188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnb7w\" (UniqueName: \"kubernetes.io/projected/88e26084-1f0d-4929-b88c-485fd03652d6-kube-api-access-wnb7w\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.362227 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.362175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:59.426087 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.426048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.426258 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.426128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e26084-1f0d-4929-b88c-485fd03652d6-openshift-service-ca\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.426258 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.426156    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnb7w\" (UniqueName: \"kubernetes.io/projected/88e26084-1f0d-4929-b88c-485fd03652d6-kube-api-access-wnb7w\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.426258 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:14:59.426201    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-serving-cert: secret "maas-api-e2e-ait-0e390b4a-serving-cert" not found
Jul 24 23:14:59.426413 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:14:59.426282    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls podName:88e26084-1f0d-4929-b88c-485fd03652d6 nodeName:}" failed. No retries permitted until 2026-07-24 23:14:59.92625915 +0000 UTC m=+1716.559538083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls") pod "maas-api-e2e-ait-0e390b4a-86555599b8-2z2km" (UID: "88e26084-1f0d-4929-b88c-485fd03652d6") : secret "maas-api-e2e-ait-0e390b4a-serving-cert" not found
Jul 24 23:14:59.426989 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.426963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e26084-1f0d-4929-b88c-485fd03652d6-openshift-service-ca\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.435794 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.435758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnb7w\" (UniqueName: \"kubernetes.io/projected/88e26084-1f0d-4929-b88c-485fd03652d6-kube-api-access-wnb7w\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.731074 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.731030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:59.833467 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.833429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:59.886480 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.886439    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"]
Jul 24 23:14:59.891344 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.891312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:14:59.893873 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.893847    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e390b4a\""
Jul 24 23:14:59.898540 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.898496    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"]
Jul 24 23:14:59.901018 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.900943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:14:59.930506 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.930464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbs8\" (UniqueName: \"kubernetes.io/projected/01867401-72c5-4fc8-b0e6-1041db98bf7d-kube-api-access-8zbs8\") pod \"payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74\" (UID: \"01867401-72c5-4fc8-b0e6-1041db98bf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:14:59.930727 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.930625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.930727 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.930673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01867401-72c5-4fc8-b0e6-1041db98bf7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74\" (UID: \"01867401-72c5-4fc8-b0e6-1041db98bf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:14:59.931238 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.931217    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"]
Jul 24 23:14:59.933173 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.933155    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls\") pod \"maas-api-e2e-ait-0e390b4a-86555599b8-2z2km\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:14:59.934508 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.934490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:14:59.938088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.938066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e390b4a-dockercfg-rxp7s\""
Jul 24 23:14:59.952030 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:14:59.951993    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"]
Jul 24 23:15:00.031347 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.031264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01867401-72c5-4fc8-b0e6-1041db98bf7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74\" (UID: \"01867401-72c5-4fc8-b0e6-1041db98bf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:00.031347 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.031307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbs8\" (UniqueName: \"kubernetes.io/projected/01867401-72c5-4fc8-b0e6-1041db98bf7d-kube-api-access-8zbs8\") pod \"payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74\" (UID: \"01867401-72c5-4fc8-b0e6-1041db98bf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:00.031560 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.031359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26qpk\" (UniqueName: \"kubernetes.io/projected/71ec7fa7-7557-4ba1-bb30-5701228e091f-kube-api-access-26qpk\") pod \"payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk\" (UID: \"71ec7fa7-7557-4ba1-bb30-5701228e091f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:00.031560 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.031380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec7fa7-7557-4ba1-bb30-5701228e091f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk\" (UID: \"71ec7fa7-7557-4ba1-bb30-5701228e091f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:00.031999 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.031979    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01867401-72c5-4fc8-b0e6-1041db98bf7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74\" (UID: \"01867401-72c5-4fc8-b0e6-1041db98bf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:00.046819 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.046788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbs8\" (UniqueName: \"kubernetes.io/projected/01867401-72c5-4fc8-b0e6-1041db98bf7d-kube-api-access-8zbs8\") pod \"payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74\" (UID: \"01867401-72c5-4fc8-b0e6-1041db98bf7d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:00.130984 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.130951    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545"]
Jul 24 23:15:00.132647 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.132612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26qpk\" (UniqueName: \"kubernetes.io/projected/71ec7fa7-7557-4ba1-bb30-5701228e091f-kube-api-access-26qpk\") pod \"payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk\" (UID: \"71ec7fa7-7557-4ba1-bb30-5701228e091f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:00.132815 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.132673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec7fa7-7557-4ba1-bb30-5701228e091f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk\" (UID: \"71ec7fa7-7557-4ba1-bb30-5701228e091f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:00.133376 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.133356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec7fa7-7557-4ba1-bb30-5701228e091f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk\" (UID: \"71ec7fa7-7557-4ba1-bb30-5701228e091f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:00.134734 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.134691    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"]
Jul 24 23:15:00.134836 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.134800    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545"
Jul 24 23:15:00.137204 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.137180    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:15:00.137756 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.137740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"
Jul 24 23:15:00.144727 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.144694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545"]
Jul 24 23:15:00.146072 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.146045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26qpk\" (UniqueName: \"kubernetes.io/projected/71ec7fa7-7557-4ba1-bb30-5701228e091f-kube-api-access-26qpk\") pod \"payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk\" (UID: \"71ec7fa7-7557-4ba1-bb30-5701228e091f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:00.146893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.146870    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"]
Jul 24 23:15:00.157216 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.157187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545"
Jul 24 23:15:00.164115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.164090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"
Jul 24 23:15:00.203921 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.203760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:00.210936 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.210885    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:15:00.249318 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.249119    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:00.340498 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.338833    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545"]
Jul 24 23:15:00.345707 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:00.345658    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d640af8_5143_4b0d_87fc_9bc1657fca00.slice/crio-fbf3ef8d7413474e3e7cd9ae6559f51259ec4b1c002edc90ed9cf8395bc572f5 WatchSource:0}: Error finding container fbf3ef8d7413474e3e7cd9ae6559f51259ec4b1c002edc90ed9cf8395bc572f5: Status 404 returned error can't find the container with id fbf3ef8d7413474e3e7cd9ae6559f51259ec4b1c002edc90ed9cf8395bc572f5
Jul 24 23:15:00.374804 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.374731    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"]
Jul 24 23:15:00.376842 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.376780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545" event={"ID":"2d640af8-5143-4b0d-87fc-9bc1657fca00","Type":"ContainerStarted","Data":"fbf3ef8d7413474e3e7cd9ae6559f51259ec4b1c002edc90ed9cf8395bc572f5"}
Jul 24 23:15:00.380161 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:00.380063    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55038ac_f736_4cdd_9b61_6f0f652bdb76.slice/crio-c916ec13125aeb6b462187f16abf6050c5564efcc439e6153032b736732683a0 WatchSource:0}: Error finding container c916ec13125aeb6b462187f16abf6050c5564efcc439e6153032b736732683a0: Status 404 returned error can't find the container with id c916ec13125aeb6b462187f16abf6050c5564efcc439e6153032b736732683a0
Jul 24 23:15:00.414815 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.414719    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"]
Jul 24 23:15:00.420197 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:00.420147    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e26084_1f0d_4929_b88c_485fd03652d6.slice/crio-8b5493fc75b74ec3e8c0783f99d30a0f02f6b3826f8d18e4693a88ae801fa3a1 WatchSource:0}: Error finding container 8b5493fc75b74ec3e8c0783f99d30a0f02f6b3826f8d18e4693a88ae801fa3a1: Status 404 returned error can't find the container with id 8b5493fc75b74ec3e8c0783f99d30a0f02f6b3826f8d18e4693a88ae801fa3a1
Jul 24 23:15:00.422688 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.422666    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 23:15:00.443905 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.443759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"]
Jul 24 23:15:00.445901 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:00.445876    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01867401_72c5_4fc8_b0e6_1041db98bf7d.slice/crio-8350304069ab614bc05d06f50ec0de2436345babb65cca5b5a0d216dc2a95e50 WatchSource:0}: Error finding container 8350304069ab614bc05d06f50ec0de2436345babb65cca5b5a0d216dc2a95e50: Status 404 returned error can't find the container with id 8350304069ab614bc05d06f50ec0de2436345babb65cca5b5a0d216dc2a95e50
Jul 24 23:15:00.467973 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:00.467949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"]
Jul 24 23:15:00.470432 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:00.470404    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ec7fa7_7557_4ba1_bb30_5701228e091f.slice/crio-746e944b573074d05201421aeeccfb7f6fd791732b9d5e6e47d771c63e137619 WatchSource:0}: Error finding container 746e944b573074d05201421aeeccfb7f6fd791732b9d5e6e47d771c63e137619: Status 404 returned error can't find the container with id 746e944b573074d05201421aeeccfb7f6fd791732b9d5e6e47d771c63e137619
Jul 24 23:15:01.395781 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.395690    2563 generic.go:356] "Generic (PLEG): container finished" podID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerID="17feca401e34657aaae464bffc47438a4ec076049ea3fd4b8b123901037fd676" exitCode=7
Jul 24 23:15:01.396239 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.395805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" event={"ID":"f55038ac-f736-4cdd-9b61-6f0f652bdb76","Type":"ContainerDied","Data":"17feca401e34657aaae464bffc47438a4ec076049ea3fd4b8b123901037fd676"}
Jul 24 23:15:01.396239 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.395856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" event={"ID":"f55038ac-f736-4cdd-9b61-6f0f652bdb76","Type":"ContainerStarted","Data":"c916ec13125aeb6b462187f16abf6050c5564efcc439e6153032b736732683a0"}
Jul 24 23:15:01.396239 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.396147    2563 scope.go:117] "RemoveContainer" containerID="17feca401e34657aaae464bffc47438a4ec076049ea3fd4b8b123901037fd676"
Jul 24 23:15:01.401136 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.401104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk" event={"ID":"71ec7fa7-7557-4ba1-bb30-5701228e091f","Type":"ContainerStarted","Data":"746e944b573074d05201421aeeccfb7f6fd791732b9d5e6e47d771c63e137619"}
Jul 24 23:15:01.415591 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.415343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545" event={"ID":"2d640af8-5143-4b0d-87fc-9bc1657fca00","Type":"ContainerStarted","Data":"a9c26c7d9aa63535a4ebcdd6dca7e57475c31e01e0d70682d6c75122cdf1efe5"}
Jul 24 23:15:01.420282 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.420160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km" event={"ID":"88e26084-1f0d-4929-b88c-485fd03652d6","Type":"ContainerStarted","Data":"8b5493fc75b74ec3e8c0783f99d30a0f02f6b3826f8d18e4693a88ae801fa3a1"}
Jul 24 23:15:01.433173 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.430792    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545" podStartSLOduration=1.430764818 podStartE2EDuration="1.430764818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:01.429822594 +0000 UTC m=+1718.063101536" watchObservedRunningTime="2026-07-24 23:15:01.430764818 +0000 UTC m=+1718.064043761"
Jul 24 23:15:01.436692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:01.436637    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74" event={"ID":"01867401-72c5-4fc8-b0e6-1041db98bf7d","Type":"ContainerStarted","Data":"8350304069ab614bc05d06f50ec0de2436345babb65cca5b5a0d216dc2a95e50"}
Jul 24 23:15:02.444728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:02.444686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" event={"ID":"f55038ac-f736-4cdd-9b61-6f0f652bdb76","Type":"ContainerStarted","Data":"07c2f3088e799a64a23d51428365f19a6b0b36af7b8166741f64a085abf7bb62"}
Jul 24 23:15:02.446607 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:02.446575    2563 generic.go:356] "Generic (PLEG): container finished" podID="2d640af8-5143-4b0d-87fc-9bc1657fca00" containerID="a9c26c7d9aa63535a4ebcdd6dca7e57475c31e01e0d70682d6c75122cdf1efe5" exitCode=0
Jul 24 23:15:02.446765 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:02.446623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545" event={"ID":"2d640af8-5143-4b0d-87fc-9bc1657fca00","Type":"ContainerDied","Data":"a9c26c7d9aa63535a4ebcdd6dca7e57475c31e01e0d70682d6c75122cdf1efe5"}
Jul 24 23:15:02.459744 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:02.459644    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" podStartSLOduration=2.459627019 podStartE2EDuration="2.459627019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:02.458949502 +0000 UTC m=+1719.092228442" watchObservedRunningTime="2026-07-24 23:15:02.459627019 +0000 UTC m=+1719.092905936"
Jul 24 23:15:03.454054 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:03.454010    2563 generic.go:356] "Generic (PLEG): container finished" podID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerID="07c2f3088e799a64a23d51428365f19a6b0b36af7b8166741f64a085abf7bb62" exitCode=7
Jul 24 23:15:03.454534 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:03.454097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" event={"ID":"f55038ac-f736-4cdd-9b61-6f0f652bdb76","Type":"ContainerDied","Data":"07c2f3088e799a64a23d51428365f19a6b0b36af7b8166741f64a085abf7bb62"}
Jul 24 23:15:03.454534 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:03.454174    2563 scope.go:117] "RemoveContainer" containerID="17feca401e34657aaae464bffc47438a4ec076049ea3fd4b8b123901037fd676"
Jul 24 23:15:03.454534 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:03.454268    2563 scope.go:117] "RemoveContainer" containerID="07c2f3088e799a64a23d51428365f19a6b0b36af7b8166741f64a085abf7bb62"
Jul 24 23:15:03.454676 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:15:03.454550    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9_odh-ai-gateway-infra(f55038ac-f736-4cdd-9b61-6f0f652bdb76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76"
Jul 24 23:15:03.622780 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:03.622757    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545"
Jul 24 23:15:04.460694 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.460660    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545"
Jul 24 23:15:04.461175 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.460668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748915-jx545" event={"ID":"2d640af8-5143-4b0d-87fc-9bc1657fca00","Type":"ContainerDied","Data":"fbf3ef8d7413474e3e7cd9ae6559f51259ec4b1c002edc90ed9cf8395bc572f5"}
Jul 24 23:15:04.461175 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.460786    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbf3ef8d7413474e3e7cd9ae6559f51259ec4b1c002edc90ed9cf8395bc572f5"
Jul 24 23:15:04.462340 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.462316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km" event={"ID":"88e26084-1f0d-4929-b88c-485fd03652d6","Type":"ContainerStarted","Data":"a78a103d51be64a5d86b341b5c2c78b37e3a31ce44464ef613608910cb2afb3e"}
Jul 24 23:15:04.462470 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.462435    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:15:04.464328 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.464299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74" event={"ID":"01867401-72c5-4fc8-b0e6-1041db98bf7d","Type":"ContainerStarted","Data":"e2d65d95ff65cbe91633974b68cc9039d24dfb64b404bbf39b69dbffe74041ab"}
Jul 24 23:15:04.464445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.464354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:04.466088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.466064    2563 scope.go:117] "RemoveContainer" containerID="07c2f3088e799a64a23d51428365f19a6b0b36af7b8166741f64a085abf7bb62"
Jul 24 23:15:04.466293 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:15:04.466274    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9_odh-ai-gateway-infra(f55038ac-f736-4cdd-9b61-6f0f652bdb76)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76"
Jul 24 23:15:04.467311 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.467289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk" event={"ID":"71ec7fa7-7557-4ba1-bb30-5701228e091f","Type":"ContainerStarted","Data":"353b91805ce571fde233d409a7171f942189d1bfd6ec4f22de4fa594a71e352f"}
Jul 24 23:15:04.467414 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.467402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:04.479988 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.479932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km" podStartSLOduration=2.317732316 podStartE2EDuration="5.479917861s" totalImagesPullingTime="3.162185545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:04.478845941 +0000 UTC m=+1721.112124910" watchObservedRunningTime="2026-07-24 23:15:04.479917861 +0000 UTC m=+1721.113196801"
Jul 24 23:15:04.493797 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.493731    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74" podStartSLOduration=2.401766802 podStartE2EDuration="5.493717383s" totalImagesPullingTime="3.091950581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:04.493144968 +0000 UTC m=+1721.126423907" watchObservedRunningTime="2026-07-24 23:15:04.493717383 +0000 UTC m=+1721.126996322"
Jul 24 23:15:04.508584 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.508510    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk" podStartSLOduration=2.405552071 podStartE2EDuration="5.508491315s" totalImagesPullingTime="3.102939244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:04.507566429 +0000 UTC m=+1721.140845370" watchObservedRunningTime="2026-07-24 23:15:04.508491315 +0000 UTC m=+1721.141770272"
Jul 24 23:15:04.633041 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:04.633002    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:15:10.477582 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:10.477541    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:15:15.474682 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:15.474648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:15.475115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:15.474732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:17.835834 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:17.835805    2563 scope.go:117] "RemoveContainer" containerID="07c2f3088e799a64a23d51428365f19a6b0b36af7b8166741f64a085abf7bb62"
Jul 24 23:15:18.537310 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:18.537274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" event={"ID":"f55038ac-f736-4cdd-9b61-6f0f652bdb76","Type":"ContainerStarted","Data":"3f342a74ba1a5e84380a76d80c8abedb8f7e13cc054b01879bbb572f41943c8e"}
Jul 24 23:15:19.548413 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:19.548377    2563 generic.go:356] "Generic (PLEG): container finished" podID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerID="3f342a74ba1a5e84380a76d80c8abedb8f7e13cc054b01879bbb572f41943c8e" exitCode=0
Jul 24 23:15:19.548853 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:19.548436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" event={"ID":"f55038ac-f736-4cdd-9b61-6f0f652bdb76","Type":"ContainerDied","Data":"3f342a74ba1a5e84380a76d80c8abedb8f7e13cc054b01879bbb572f41943c8e"}
Jul 24 23:15:19.548853 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:19.548530    2563 scope.go:117] "RemoveContainer" containerID="07c2f3088e799a64a23d51428365f19a6b0b36af7b8166741f64a085abf7bb62"
Jul 24 23:15:19.561563 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:19.561533    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"]
Jul 24 23:15:20.689211 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:20.689182    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"
Jul 24 23:15:21.560830 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:21.560804    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"
Jul 24 23:15:21.560830 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:21.560815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9" event={"ID":"f55038ac-f736-4cdd-9b61-6f0f652bdb76","Type":"ContainerDied","Data":"c916ec13125aeb6b462187f16abf6050c5564efcc439e6153032b736732683a0"}
Jul 24 23:15:21.561133 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:21.560868    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c916ec13125aeb6b462187f16abf6050c5564efcc439e6153032b736732683a0"
Jul 24 23:15:21.591416 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:21.591384    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"]
Jul 24 23:15:21.597087 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:21.597058    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-0e390b4a-29748915-v5mg9"]
Jul 24 23:15:21.841031 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:21.840951    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" path="/var/lib/kubelet/pods/f55038ac-f736-4cdd-9b61-6f0f652bdb76/volumes"
Jul 24 23:15:24.783669 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.783631    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"]
Jul 24 23:15:24.784053 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.783997    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:24.784053 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784009    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:24.784053 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784019    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d640af8-5143-4b0d-87fc-9bc1657fca00" containerName="cleanup"
Jul 24 23:15:24.784053 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784025    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d640af8-5143-4b0d-87fc-9bc1657fca00" containerName="cleanup"
Jul 24 23:15:24.784053 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784034    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:24.784053 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784040    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:24.784283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784096    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:24.784283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784103    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:24.784283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784111    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:24.784283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.784117    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d640af8-5143-4b0d-87fc-9bc1657fca00" containerName="cleanup"
Jul 24 23:15:24.787242 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.787224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"
Jul 24 23:15:24.790015 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.789996    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:15:24.794746 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.794534    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"]
Jul 24 23:15:24.863257 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.863216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e860d0b5-bd5c-4097-8ba4-56e867e98cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e390b4a-vc929\" (UID: \"e860d0b5-bd5c-4097-8ba4-56e867e98cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"
Jul 24 23:15:24.964565 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.964506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e860d0b5-bd5c-4097-8ba4-56e867e98cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e390b4a-vc929\" (UID: \"e860d0b5-bd5c-4097-8ba4-56e867e98cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"
Jul 24 23:15:24.965182 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:24.965161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e860d0b5-bd5c-4097-8ba4-56e867e98cc0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e390b4a-vc929\" (UID: \"e860d0b5-bd5c-4097-8ba4-56e867e98cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"
Jul 24 23:15:25.098757 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:25.098726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"
Jul 24 23:15:25.236567 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:25.236537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"]
Jul 24 23:15:25.238555 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:25.238514    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode860d0b5_bd5c_4097_8ba4_56e867e98cc0.slice/crio-1b0b64ff5b45d1359cc2fd2ff03cc20c77f587b524511e3edbbbb2fe011e69ba WatchSource:0}: Error finding container 1b0b64ff5b45d1359cc2fd2ff03cc20c77f587b524511e3edbbbb2fe011e69ba: Status 404 returned error can't find the container with id 1b0b64ff5b45d1359cc2fd2ff03cc20c77f587b524511e3edbbbb2fe011e69ba
Jul 24 23:15:25.580834 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:25.580797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929" event={"ID":"e860d0b5-bd5c-4097-8ba4-56e867e98cc0","Type":"ContainerStarted","Data":"14c988e6bd9b03e2cbe07bee3ffa136f5412e9fa4f23766b0c0aada1f85f43f8"}
Jul 24 23:15:25.580992 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:25.580849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929" event={"ID":"e860d0b5-bd5c-4097-8ba4-56e867e98cc0","Type":"ContainerStarted","Data":"1b0b64ff5b45d1359cc2fd2ff03cc20c77f587b524511e3edbbbb2fe011e69ba"}
Jul 24 23:15:25.595510 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:25.595453    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929" podStartSLOduration=1.595434106 podStartE2EDuration="1.595434106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:25.594874289 +0000 UTC m=+1742.228153229" watchObservedRunningTime="2026-07-24 23:15:25.595434106 +0000 UTC m=+1742.228713045"
Jul 24 23:15:26.587371 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:26.587337    2563 generic.go:356] "Generic (PLEG): container finished" podID="e860d0b5-bd5c-4097-8ba4-56e867e98cc0" containerID="14c988e6bd9b03e2cbe07bee3ffa136f5412e9fa4f23766b0c0aada1f85f43f8" exitCode=0
Jul 24 23:15:26.587837 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:26.587434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929" event={"ID":"e860d0b5-bd5c-4097-8ba4-56e867e98cc0","Type":"ContainerDied","Data":"14c988e6bd9b03e2cbe07bee3ffa136f5412e9fa4f23766b0c0aada1f85f43f8"}
Jul 24 23:15:27.730012 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:27.729981    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"
Jul 24 23:15:27.891567 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:27.891444    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e860d0b5-bd5c-4097-8ba4-56e867e98cc0-maas-api-service-ca\") pod \"e860d0b5-bd5c-4097-8ba4-56e867e98cc0\" (UID: \"e860d0b5-bd5c-4097-8ba4-56e867e98cc0\") "
Jul 24 23:15:27.891852 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:27.891827    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e860d0b5-bd5c-4097-8ba4-56e867e98cc0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e860d0b5-bd5c-4097-8ba4-56e867e98cc0" (UID: "e860d0b5-bd5c-4097-8ba4-56e867e98cc0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:15:27.992944 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:27.992907    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e860d0b5-bd5c-4097-8ba4-56e867e98cc0-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:15:28.598466 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:28.598424    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"
Jul 24 23:15:28.598466 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:28.598459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929" event={"ID":"e860d0b5-bd5c-4097-8ba4-56e867e98cc0","Type":"ContainerDied","Data":"1b0b64ff5b45d1359cc2fd2ff03cc20c77f587b524511e3edbbbb2fe011e69ba"}
Jul 24 23:15:28.598720 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:28.598514    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0b64ff5b45d1359cc2fd2ff03cc20c77f587b524511e3edbbbb2fe011e69ba"
Jul 24 23:15:35.346741 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.346701    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"]
Jul 24 23:15:35.347645 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.347589    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km" podUID="88e26084-1f0d-4929-b88c-485fd03652d6" containerName="maas-api" containerID="cri-o://a78a103d51be64a5d86b341b5c2c78b37e3a31ce44464ef613608910cb2afb3e" gracePeriod=30
Jul 24 23:15:35.373738 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.373701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:35.404176 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.404081    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"]
Jul 24 23:15:35.404435 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.404336    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk" podUID="71ec7fa7-7557-4ba1-bb30-5701228e091f" containerName="payload-processing" containerID="cri-o://353b91805ce571fde233d409a7171f942189d1bfd6ec4f22de4fa594a71e352f" gracePeriod=30
Jul 24 23:15:35.407178 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.407143    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"]
Jul 24 23:15:35.407456 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.407405    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74" podUID="01867401-72c5-4fc8-b0e6-1041db98bf7d" containerName="payload-pre-processing" containerID="cri-o://e2d65d95ff65cbe91633974b68cc9039d24dfb64b404bbf39b69dbffe74041ab" gracePeriod=30
Jul 24 23:15:35.446399 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.446354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:35.472893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.472854    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk" podUID="71ec7fa7-7557-4ba1-bb30-5701228e091f" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.33:9004: connect: connection refused"
Jul 24 23:15:35.472893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.472878    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km" podUID="88e26084-1f0d-4929-b88c-485fd03652d6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.31:8443/health\": dial tcp 10.134.0.31:8443: connect: connection refused"
Jul 24 23:15:35.473105 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.472852    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74" podUID="01867401-72c5-4fc8-b0e6-1041db98bf7d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.32:9004: connect: connection refused"
Jul 24 23:15:35.514252 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.514196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:35.633311 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.633225    2563 generic.go:356] "Generic (PLEG): container finished" podID="71ec7fa7-7557-4ba1-bb30-5701228e091f" containerID="353b91805ce571fde233d409a7171f942189d1bfd6ec4f22de4fa594a71e352f" exitCode=0
Jul 24 23:15:35.633466 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.633294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk" event={"ID":"71ec7fa7-7557-4ba1-bb30-5701228e091f","Type":"ContainerDied","Data":"353b91805ce571fde233d409a7171f942189d1bfd6ec4f22de4fa594a71e352f"}
Jul 24 23:15:35.635360 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.635328    2563 generic.go:356] "Generic (PLEG): container finished" podID="88e26084-1f0d-4929-b88c-485fd03652d6" containerID="a78a103d51be64a5d86b341b5c2c78b37e3a31ce44464ef613608910cb2afb3e" exitCode=0
Jul 24 23:15:35.635474 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.635440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km" event={"ID":"88e26084-1f0d-4929-b88c-485fd03652d6","Type":"ContainerDied","Data":"a78a103d51be64a5d86b341b5c2c78b37e3a31ce44464ef613608910cb2afb3e"}
Jul 24 23:15:35.637713 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.637550    2563 generic.go:356] "Generic (PLEG): container finished" podID="01867401-72c5-4fc8-b0e6-1041db98bf7d" containerID="e2d65d95ff65cbe91633974b68cc9039d24dfb64b404bbf39b69dbffe74041ab" exitCode=0
Jul 24 23:15:35.637713 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.637648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74" event={"ID":"01867401-72c5-4fc8-b0e6-1041db98bf7d","Type":"ContainerDied","Data":"e2d65d95ff65cbe91633974b68cc9039d24dfb64b404bbf39b69dbffe74041ab"}
Jul 24 23:15:35.670934 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.670900    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:15:35.702877 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.702848    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:35.711058 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.711026    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:35.864533 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.864480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01867401-72c5-4fc8-b0e6-1041db98bf7d-plugins-config-volume\") pod \"01867401-72c5-4fc8-b0e6-1041db98bf7d\" (UID: \"01867401-72c5-4fc8-b0e6-1041db98bf7d\") "
Jul 24 23:15:35.864728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.864559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zbs8\" (UniqueName: \"kubernetes.io/projected/01867401-72c5-4fc8-b0e6-1041db98bf7d-kube-api-access-8zbs8\") pod \"01867401-72c5-4fc8-b0e6-1041db98bf7d\" (UID: \"01867401-72c5-4fc8-b0e6-1041db98bf7d\") "
Jul 24 23:15:35.864728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.864606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnb7w\" (UniqueName: \"kubernetes.io/projected/88e26084-1f0d-4929-b88c-485fd03652d6-kube-api-access-wnb7w\") pod \"88e26084-1f0d-4929-b88c-485fd03652d6\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") "
Jul 24 23:15:35.864728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.864686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls\") pod \"88e26084-1f0d-4929-b88c-485fd03652d6\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") "
Jul 24 23:15:35.864728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.864719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26qpk\" (UniqueName: \"kubernetes.io/projected/71ec7fa7-7557-4ba1-bb30-5701228e091f-kube-api-access-26qpk\") pod \"71ec7fa7-7557-4ba1-bb30-5701228e091f\" (UID: \"71ec7fa7-7557-4ba1-bb30-5701228e091f\") "
Jul 24 23:15:35.864945 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.864742    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec7fa7-7557-4ba1-bb30-5701228e091f-plugins-config-volume\") pod \"71ec7fa7-7557-4ba1-bb30-5701228e091f\" (UID: \"71ec7fa7-7557-4ba1-bb30-5701228e091f\") "
Jul 24 23:15:35.864998 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.864939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e26084-1f0d-4929-b88c-485fd03652d6-openshift-service-ca\") pod \"88e26084-1f0d-4929-b88c-485fd03652d6\" (UID: \"88e26084-1f0d-4929-b88c-485fd03652d6\") "
Jul 24 23:15:35.865445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.865079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01867401-72c5-4fc8-b0e6-1041db98bf7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01867401-72c5-4fc8-b0e6-1041db98bf7d" (UID: "01867401-72c5-4fc8-b0e6-1041db98bf7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:15:35.865445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.865121    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ec7fa7-7557-4ba1-bb30-5701228e091f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71ec7fa7-7557-4ba1-bb30-5701228e091f" (UID: "71ec7fa7-7557-4ba1-bb30-5701228e091f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:15:35.865445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.865389    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ec7fa7-7557-4ba1-bb30-5701228e091f-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:15:35.865445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.865403    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e26084-1f0d-4929-b88c-485fd03652d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "88e26084-1f0d-4929-b88c-485fd03652d6" (UID: "88e26084-1f0d-4929-b88c-485fd03652d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:15:35.865445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.865410    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01867401-72c5-4fc8-b0e6-1041db98bf7d-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:15:35.867032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.867008    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "88e26084-1f0d-4929-b88c-485fd03652d6" (UID: "88e26084-1f0d-4929-b88c-485fd03652d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:15:35.867132 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.867111    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e26084-1f0d-4929-b88c-485fd03652d6-kube-api-access-wnb7w" (OuterVolumeSpecName: "kube-api-access-wnb7w") pod "88e26084-1f0d-4929-b88c-485fd03652d6" (UID: "88e26084-1f0d-4929-b88c-485fd03652d6"). InnerVolumeSpecName "kube-api-access-wnb7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:15:35.867247 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.867230    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ec7fa7-7557-4ba1-bb30-5701228e091f-kube-api-access-26qpk" (OuterVolumeSpecName: "kube-api-access-26qpk") pod "71ec7fa7-7557-4ba1-bb30-5701228e091f" (UID: "71ec7fa7-7557-4ba1-bb30-5701228e091f"). InnerVolumeSpecName "kube-api-access-26qpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:15:35.867247 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.867236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01867401-72c5-4fc8-b0e6-1041db98bf7d-kube-api-access-8zbs8" (OuterVolumeSpecName: "kube-api-access-8zbs8") pod "01867401-72c5-4fc8-b0e6-1041db98bf7d" (UID: "01867401-72c5-4fc8-b0e6-1041db98bf7d"). InnerVolumeSpecName "kube-api-access-8zbs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:15:35.966731 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.966635    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e26084-1f0d-4929-b88c-485fd03652d6-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:15:35.966731 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.966674    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbs8\" (UniqueName: \"kubernetes.io/projected/01867401-72c5-4fc8-b0e6-1041db98bf7d-kube-api-access-8zbs8\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:15:35.966731 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.966690    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnb7w\" (UniqueName: \"kubernetes.io/projected/88e26084-1f0d-4929-b88c-485fd03652d6-kube-api-access-wnb7w\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:15:35.966731 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.966704    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88e26084-1f0d-4929-b88c-485fd03652d6-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:15:35.966731 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:35.966717    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26qpk\" (UniqueName: \"kubernetes.io/projected/71ec7fa7-7557-4ba1-bb30-5701228e091f-kube-api-access-26qpk\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:15:36.644473 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.644380    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"
Jul 24 23:15:36.644473 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.644388    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km" event={"ID":"88e26084-1f0d-4929-b88c-485fd03652d6","Type":"ContainerDied","Data":"8b5493fc75b74ec3e8c0783f99d30a0f02f6b3826f8d18e4693a88ae801fa3a1"}
Jul 24 23:15:36.644473 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.644447    2563 scope.go:117] "RemoveContainer" containerID="a78a103d51be64a5d86b341b5c2c78b37e3a31ce44464ef613608910cb2afb3e"
Jul 24 23:15:36.646025 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.646003    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"
Jul 24 23:15:36.646106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.646019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74" event={"ID":"01867401-72c5-4fc8-b0e6-1041db98bf7d","Type":"ContainerDied","Data":"8350304069ab614bc05d06f50ec0de2436345babb65cca5b5a0d216dc2a95e50"}
Jul 24 23:15:36.647355 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.647319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk" event={"ID":"71ec7fa7-7557-4ba1-bb30-5701228e091f","Type":"ContainerDied","Data":"746e944b573074d05201421aeeccfb7f6fd791732b9d5e6e47d771c63e137619"}
Jul 24 23:15:36.647355 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.647343    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"
Jul 24 23:15:36.654579 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.654489    2563 scope.go:117] "RemoveContainer" containerID="e2d65d95ff65cbe91633974b68cc9039d24dfb64b404bbf39b69dbffe74041ab"
Jul 24 23:15:36.665178 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.665156    2563 scope.go:117] "RemoveContainer" containerID="353b91805ce571fde233d409a7171f942189d1bfd6ec4f22de4fa594a71e352f"
Jul 24 23:15:36.673491 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.673454    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"]
Jul 24 23:15:36.677004 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.676975    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e390b4a-5dfd8465bd-n2r74"]
Jul 24 23:15:36.688168 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.688130    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"]
Jul 24 23:15:36.691533 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.691488    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e390b4a-59b8fdf848-c68dk"]
Jul 24 23:15:36.701534 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.701486    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"]
Jul 24 23:15:36.704559 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:36.704509    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e390b4a-86555599b8-2z2km"]
Jul 24 23:15:37.841704 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:37.841671    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01867401-72c5-4fc8-b0e6-1041db98bf7d" path="/var/lib/kubelet/pods/01867401-72c5-4fc8-b0e6-1041db98bf7d/volumes"
Jul 24 23:15:37.842095 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:37.842033    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71ec7fa7-7557-4ba1-bb30-5701228e091f" path="/var/lib/kubelet/pods/71ec7fa7-7557-4ba1-bb30-5701228e091f/volumes"
Jul 24 23:15:37.842372 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:37.842359    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e26084-1f0d-4929-b88c-485fd03652d6" path="/var/lib/kubelet/pods/88e26084-1f0d-4929-b88c-485fd03652d6/volumes"
Jul 24 23:15:38.424891 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:38.424857    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:15:39.860945 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:39.860898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:39.861951 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:39.861928    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"]
Jul 24 23:15:39.864285 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:39.864260    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e390b4a-vc929"]
Jul 24 23:15:40.026179 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:40.026141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:40.298093 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:40.298006    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:40.508122 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:40.508087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:40.604253 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:40.604222    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:40.704224 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:40.704192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:41.022557 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:41.022431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:41.282996 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:41.282913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:41.842002 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:41.841960    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e860d0b5-bd5c-4097-8ba4-56e867e98cc0" path="/var/lib/kubelet/pods/e860d0b5-bd5c-4097-8ba4-56e867e98cc0/volumes"
Jul 24 23:15:43.678249 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.678214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:43.755934 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.755856    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"]
Jul 24 23:15:43.756501 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756469    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01867401-72c5-4fc8-b0e6-1041db98bf7d" containerName="payload-pre-processing"
Jul 24 23:15:43.756501 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756493    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="01867401-72c5-4fc8-b0e6-1041db98bf7d" containerName="payload-pre-processing"
Jul 24 23:15:43.756656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756514    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e26084-1f0d-4929-b88c-485fd03652d6" containerName="maas-api"
Jul 24 23:15:43.756656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756541    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e26084-1f0d-4929-b88c-485fd03652d6" containerName="maas-api"
Jul 24 23:15:43.756656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756562    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71ec7fa7-7557-4ba1-bb30-5701228e091f" containerName="payload-processing"
Jul 24 23:15:43.756656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756572    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71ec7fa7-7557-4ba1-bb30-5701228e091f" containerName="payload-processing"
Jul 24 23:15:43.756656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756591    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e860d0b5-bd5c-4097-8ba4-56e867e98cc0" containerName="revoke-keys"
Jul 24 23:15:43.756656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756600    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e860d0b5-bd5c-4097-8ba4-56e867e98cc0" containerName="revoke-keys"
Jul 24 23:15:43.756656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756614    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:43.756656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756622    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55038ac-f736-4cdd-9b61-6f0f652bdb76" containerName="cleanup"
Jul 24 23:15:43.757030 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756709    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71ec7fa7-7557-4ba1-bb30-5701228e091f" containerName="payload-processing"
Jul 24 23:15:43.757030 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756720    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e26084-1f0d-4929-b88c-485fd03652d6" containerName="maas-api"
Jul 24 23:15:43.757030 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756733    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e860d0b5-bd5c-4097-8ba4-56e867e98cc0" containerName="revoke-keys"
Jul 24 23:15:43.757030 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.756743    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="01867401-72c5-4fc8-b0e6-1041db98bf7d" containerName="payload-pre-processing"
Jul 24 23:15:43.762887 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.762861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:43.765490 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.765434    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f5e9b15b-serving-cert\""
Jul 24 23:15:43.765768 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.765748    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:15:43.765866 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.765788    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:15:43.766786 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.766765    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:15:43.769351 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.769247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"]
Jul 24 23:15:43.792658 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.792617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:43.833956 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.833906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wb7\" (UniqueName: \"kubernetes.io/projected/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-kube-api-access-b9wb7\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:43.834154 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.833983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:43.834154 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.834055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:43.935014 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.934912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:43.935014 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.934988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:43.935291 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.935037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wb7\" (UniqueName: \"kubernetes.io/projected/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-kube-api-access-b9wb7\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:43.935291 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:15:43.935114    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-serving-cert: secret "maas-api-e2e-migrate-f5e9b15b-serving-cert" not found
Jul 24 23:15:43.935291 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:15:43.935183    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls podName:0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4 nodeName:}" failed. No retries permitted until 2026-07-24 23:15:44.435161083 +0000 UTC m=+1761.068440013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls") pod "maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq" (UID: "0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4") : secret "maas-api-e2e-migrate-f5e9b15b-serving-cert" not found
Jul 24 23:15:43.935709 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.935691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-openshift-service-ca\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:43.943598 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:43.943572    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wb7\" (UniqueName: \"kubernetes.io/projected/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-kube-api-access-b9wb7\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:44.255000 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.254916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:44.390263 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.390233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:44.439883 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.439843    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:44.442425 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.442389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls\") pod \"maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:44.493708 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.493673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:15:44.494323 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.494295    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"]
Jul 24 23:15:44.496660 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.496639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:44.500611 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.500587    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f5e9b15b\""
Jul 24 23:15:44.500745 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.500693    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f5e9b15b-dockercfg-gln2h\""
Jul 24 23:15:44.511792 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.511721    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"]
Jul 24 23:15:44.530622 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.530586    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"]
Jul 24 23:15:44.532995 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.532970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:44.540392 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.540364    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569c311a-82a3-4519-87b0-2046e9b27692-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm\" (UID: \"569c311a-82a3-4519-87b0-2046e9b27692\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:44.540539 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.540399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvpgp\" (UniqueName: \"kubernetes.io/projected/569c311a-82a3-4519-87b0-2046e9b27692-kube-api-access-mvpgp\") pod \"payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm\" (UID: \"569c311a-82a3-4519-87b0-2046e9b27692\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:44.545081 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.545050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"]
Jul 24 23:15:44.640902 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.640862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d478d0c-13ea-4201-a5a9-1dfc34205439-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f\" (UID: \"3d478d0c-13ea-4201-a5a9-1dfc34205439\") " pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:44.641077 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.640982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569c311a-82a3-4519-87b0-2046e9b27692-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm\" (UID: \"569c311a-82a3-4519-87b0-2046e9b27692\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:44.641077 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.641024    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvpgp\" (UniqueName: \"kubernetes.io/projected/569c311a-82a3-4519-87b0-2046e9b27692-kube-api-access-mvpgp\") pod \"payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm\" (UID: \"569c311a-82a3-4519-87b0-2046e9b27692\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:44.641077 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.641050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbnm\" (UniqueName: \"kubernetes.io/projected/3d478d0c-13ea-4201-a5a9-1dfc34205439-kube-api-access-5qbnm\") pod \"payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f\" (UID: \"3d478d0c-13ea-4201-a5a9-1dfc34205439\") " pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:44.641759 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.641738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569c311a-82a3-4519-87b0-2046e9b27692-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm\" (UID: \"569c311a-82a3-4519-87b0-2046e9b27692\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:44.649035 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.648999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvpgp\" (UniqueName: \"kubernetes.io/projected/569c311a-82a3-4519-87b0-2046e9b27692-kube-api-access-mvpgp\") pod \"payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm\" (UID: \"569c311a-82a3-4519-87b0-2046e9b27692\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:44.676032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.675987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:44.741988 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.741934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbnm\" (UniqueName: \"kubernetes.io/projected/3d478d0c-13ea-4201-a5a9-1dfc34205439-kube-api-access-5qbnm\") pod \"payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f\" (UID: \"3d478d0c-13ea-4201-a5a9-1dfc34205439\") " pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:44.743093 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.742019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d478d0c-13ea-4201-a5a9-1dfc34205439-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f\" (UID: \"3d478d0c-13ea-4201-a5a9-1dfc34205439\") " pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:44.743093 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.742997    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d478d0c-13ea-4201-a5a9-1dfc34205439-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f\" (UID: \"3d478d0c-13ea-4201-a5a9-1dfc34205439\") " pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:44.750211 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.750185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbnm\" (UniqueName: \"kubernetes.io/projected/3d478d0c-13ea-4201-a5a9-1dfc34205439-kube-api-access-5qbnm\") pod \"payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f\" (UID: \"3d478d0c-13ea-4201-a5a9-1dfc34205439\") " pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:44.808902 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.808854    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:44.836581 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.836554    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"]
Jul 24 23:15:44.842782 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:44.842734    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a93e03c_68bf_4fd7_bfd9_2b20b310c5e4.slice/crio-1da1119a3478940341557eea81829702f184700068b8ff416e15cef073eb2a96 WatchSource:0}: Error finding container 1da1119a3478940341557eea81829702f184700068b8ff416e15cef073eb2a96: Status 404 returned error can't find the container with id 1da1119a3478940341557eea81829702f184700068b8ff416e15cef073eb2a96
Jul 24 23:15:44.843825 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.843803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:44.958174 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.958129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"]
Jul 24 23:15:44.960999 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:44.960966    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569c311a_82a3_4519_87b0_2046e9b27692.slice/crio-d1ca5dcc979cef7f5fe438fdf4ebaeaf17b8c36b3efc434bf772abd0f10f2541 WatchSource:0}: Error finding container d1ca5dcc979cef7f5fe438fdf4ebaeaf17b8c36b3efc434bf772abd0f10f2541: Status 404 returned error can't find the container with id d1ca5dcc979cef7f5fe438fdf4ebaeaf17b8c36b3efc434bf772abd0f10f2541
Jul 24 23:15:44.999515 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:44.999486    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"]
Jul 24 23:15:45.003709 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:15:45.002674    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d478d0c_13ea_4201_a5a9_1dfc34205439.slice/crio-fc2e13059ffe6af781871e44bddbb3b03068ebda33b484413334297791d9092b WatchSource:0}: Error finding container fc2e13059ffe6af781871e44bddbb3b03068ebda33b484413334297791d9092b: Status 404 returned error can't find the container with id fc2e13059ffe6af781871e44bddbb3b03068ebda33b484413334297791d9092b
Jul 24 23:15:45.695884 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.695844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq" event={"ID":"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4","Type":"ContainerStarted","Data":"2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c"}
Jul 24 23:15:45.696066 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.695905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq" event={"ID":"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4","Type":"ContainerStarted","Data":"1da1119a3478940341557eea81829702f184700068b8ff416e15cef073eb2a96"}
Jul 24 23:15:45.696066 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.695960    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:45.697431 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.697400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f" event={"ID":"3d478d0c-13ea-4201-a5a9-1dfc34205439","Type":"ContainerStarted","Data":"da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79"}
Jul 24 23:15:45.697586 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.697455    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:45.697586 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.697479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f" event={"ID":"3d478d0c-13ea-4201-a5a9-1dfc34205439","Type":"ContainerStarted","Data":"fc2e13059ffe6af781871e44bddbb3b03068ebda33b484413334297791d9092b"}
Jul 24 23:15:45.699440 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.699411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm" event={"ID":"569c311a-82a3-4519-87b0-2046e9b27692","Type":"ContainerStarted","Data":"3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0"}
Jul 24 23:15:45.699583 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.699461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm" event={"ID":"569c311a-82a3-4519-87b0-2046e9b27692","Type":"ContainerStarted","Data":"d1ca5dcc979cef7f5fe438fdf4ebaeaf17b8c36b3efc434bf772abd0f10f2541"}
Jul 24 23:15:45.699844 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.699819    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:15:45.712915 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.712862    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq" podStartSLOduration=2.4211803290000002 podStartE2EDuration="2.712826925s" totalImagesPullingTime="291.646596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:45.711419132 +0000 UTC m=+1762.344698089" watchObservedRunningTime="2026-07-24 23:15:45.712826925 +0000 UTC m=+1762.346105857"
Jul 24 23:15:45.726943 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.726882    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm" podStartSLOduration=1.7268607390000001 podStartE2EDuration="1.726860739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:45.726291419 +0000 UTC m=+1762.359570359" watchObservedRunningTime="2026-07-24 23:15:45.726860739 +0000 UTC m=+1762.360139682"
Jul 24 23:15:45.742699 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:45.742640    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f" podStartSLOduration=1.742623515 podStartE2EDuration="1.742623515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:15:45.74104185 +0000 UTC m=+1762.374320801" watchObservedRunningTime="2026-07-24 23:15:45.742623515 +0000 UTC m=+1762.375902454"
Jul 24 23:15:49.425532 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:49.425495    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:15:51.710858 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:51.710826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:15:56.707072 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:56.707043    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:15:56.707576 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:15:56.707127    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:16:08.351392 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.351359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:08.577972 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.577937    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"]
Jul 24 23:16:08.581183 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.581164    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"
Jul 24 23:16:08.583884 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.583863    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:16:08.588396 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.588369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"]
Jul 24 23:16:08.657009 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.656913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73d77e7-1f64-4262-8d49-d2a5134bc9d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq\" (UID: \"c73d77e7-1f64-4262-8d49-d2a5134bc9d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"
Jul 24 23:16:08.758117 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.758080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73d77e7-1f64-4262-8d49-d2a5134bc9d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq\" (UID: \"c73d77e7-1f64-4262-8d49-d2a5134bc9d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"
Jul 24 23:16:08.758740 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.758719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73d77e7-1f64-4262-8d49-d2a5134bc9d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq\" (UID: \"c73d77e7-1f64-4262-8d49-d2a5134bc9d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"
Jul 24 23:16:08.893614 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:08.893559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"
Jul 24 23:16:09.029768 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:09.029728    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"]
Jul 24 23:16:09.033113 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:16:09.033085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73d77e7_1f64_4262_8d49_d2a5134bc9d4.slice/crio-ff5be531ae486c6a656628bb2678b75427052c03217d26a7331c57ba4822aea6 WatchSource:0}: Error finding container ff5be531ae486c6a656628bb2678b75427052c03217d26a7331c57ba4822aea6: Status 404 returned error can't find the container with id ff5be531ae486c6a656628bb2678b75427052c03217d26a7331c57ba4822aea6
Jul 24 23:16:09.823970 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:09.823930    2563 generic.go:356] "Generic (PLEG): container finished" podID="c73d77e7-1f64-4262-8d49-d2a5134bc9d4" containerID="56db39cb409cfb04fb5be6361e7cfd86f7c0b8cd8ce8f4b05a89a37925220726" exitCode=0
Jul 24 23:16:09.824377 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:09.824003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq" event={"ID":"c73d77e7-1f64-4262-8d49-d2a5134bc9d4","Type":"ContainerDied","Data":"56db39cb409cfb04fb5be6361e7cfd86f7c0b8cd8ce8f4b05a89a37925220726"}
Jul 24 23:16:09.824377 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:09.824081    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq" event={"ID":"c73d77e7-1f64-4262-8d49-d2a5134bc9d4","Type":"ContainerStarted","Data":"ff5be531ae486c6a656628bb2678b75427052c03217d26a7331c57ba4822aea6"}
Jul 24 23:16:10.966725 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:10.966701    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"
Jul 24 23:16:11.078992 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:11.078952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73d77e7-1f64-4262-8d49-d2a5134bc9d4-maas-api-service-ca\") pod \"c73d77e7-1f64-4262-8d49-d2a5134bc9d4\" (UID: \"c73d77e7-1f64-4262-8d49-d2a5134bc9d4\") "
Jul 24 23:16:11.079329 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:11.079304    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73d77e7-1f64-4262-8d49-d2a5134bc9d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c73d77e7-1f64-4262-8d49-d2a5134bc9d4" (UID: "c73d77e7-1f64-4262-8d49-d2a5134bc9d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:16:11.179643 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:11.179605    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c73d77e7-1f64-4262-8d49-d2a5134bc9d4-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:11.835769 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:11.835743    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"
Jul 24 23:16:11.841061 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:11.841033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq" event={"ID":"c73d77e7-1f64-4262-8d49-d2a5134bc9d4","Type":"ContainerDied","Data":"ff5be531ae486c6a656628bb2678b75427052c03217d26a7331c57ba4822aea6"}
Jul 24 23:16:11.841208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:11.841077    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff5be531ae486c6a656628bb2678b75427052c03217d26a7331c57ba4822aea6"
Jul 24 23:16:18.986594 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:18.986558    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"]
Jul 24 23:16:18.987096 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:18.986823    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq" podUID="0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" containerName="maas-api" containerID="cri-o://2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c" gracePeriod=30
Jul 24 23:16:19.010384 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.010353    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:19.035356 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.035321    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"]
Jul 24 23:16:19.035656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.035631    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f" podUID="3d478d0c-13ea-4201-a5a9-1dfc34205439" containerName="payload-processing" containerID="cri-o://da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79" gracePeriod=30
Jul 24 23:16:19.043070 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.043006    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"]
Jul 24 23:16:19.043287 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.043258    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm" podUID="569c311a-82a3-4519-87b0-2046e9b27692" containerName="payload-pre-processing" containerID="cri-o://3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0" gracePeriod=30
Jul 24 23:16:19.081194 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.081161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:19.137776 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.137738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:19.317588 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.317554    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:16:19.334841 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.334808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:16:19.344835 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.344808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:16:19.364952 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.364850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvpgp\" (UniqueName: \"kubernetes.io/projected/569c311a-82a3-4519-87b0-2046e9b27692-kube-api-access-mvpgp\") pod \"569c311a-82a3-4519-87b0-2046e9b27692\" (UID: \"569c311a-82a3-4519-87b0-2046e9b27692\") "
Jul 24 23:16:19.365334 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.365310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9wb7\" (UniqueName: \"kubernetes.io/projected/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-kube-api-access-b9wb7\") pod \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") "
Jul 24 23:16:19.365446 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.365383    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d478d0c-13ea-4201-a5a9-1dfc34205439-plugins-config-volume\") pod \"3d478d0c-13ea-4201-a5a9-1dfc34205439\" (UID: \"3d478d0c-13ea-4201-a5a9-1dfc34205439\") "
Jul 24 23:16:19.365446 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.365412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569c311a-82a3-4519-87b0-2046e9b27692-plugins-config-volume\") pod \"569c311a-82a3-4519-87b0-2046e9b27692\" (UID: \"569c311a-82a3-4519-87b0-2046e9b27692\") "
Jul 24 23:16:19.365601 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.365488    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-openshift-service-ca\") pod \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") "
Jul 24 23:16:19.366127 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.366029    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls\") pod \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\" (UID: \"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4\") "
Jul 24 23:16:19.366127 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.366076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qbnm\" (UniqueName: \"kubernetes.io/projected/3d478d0c-13ea-4201-a5a9-1dfc34205439-kube-api-access-5qbnm\") pod \"3d478d0c-13ea-4201-a5a9-1dfc34205439\" (UID: \"3d478d0c-13ea-4201-a5a9-1dfc34205439\") "
Jul 24 23:16:19.367268 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.366292    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" (UID: "0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:16:19.367268 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.366804    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569c311a-82a3-4519-87b0-2046e9b27692-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "569c311a-82a3-4519-87b0-2046e9b27692" (UID: "569c311a-82a3-4519-87b0-2046e9b27692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:16:19.367268 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.366843    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d478d0c-13ea-4201-a5a9-1dfc34205439-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d478d0c-13ea-4201-a5a9-1dfc34205439" (UID: "3d478d0c-13ea-4201-a5a9-1dfc34205439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:16:19.367268 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.367101    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d478d0c-13ea-4201-a5a9-1dfc34205439-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:19.367268 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.367122    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569c311a-82a3-4519-87b0-2046e9b27692-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:19.367268 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.367135    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:19.368201 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.368173    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-kube-api-access-b9wb7" (OuterVolumeSpecName: "kube-api-access-b9wb7") pod "0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" (UID: "0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4"). InnerVolumeSpecName "kube-api-access-b9wb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:16:19.368749 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.368724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569c311a-82a3-4519-87b0-2046e9b27692-kube-api-access-mvpgp" (OuterVolumeSpecName: "kube-api-access-mvpgp") pod "569c311a-82a3-4519-87b0-2046e9b27692" (UID: "569c311a-82a3-4519-87b0-2046e9b27692"). InnerVolumeSpecName "kube-api-access-mvpgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:16:19.369495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.369474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" (UID: "0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:16:19.369930 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.369906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d478d0c-13ea-4201-a5a9-1dfc34205439-kube-api-access-5qbnm" (OuterVolumeSpecName: "kube-api-access-5qbnm") pod "3d478d0c-13ea-4201-a5a9-1dfc34205439" (UID: "3d478d0c-13ea-4201-a5a9-1dfc34205439"). InnerVolumeSpecName "kube-api-access-5qbnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:16:19.468171 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.468133    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:19.468171 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.468163    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbnm\" (UniqueName: \"kubernetes.io/projected/3d478d0c-13ea-4201-a5a9-1dfc34205439-kube-api-access-5qbnm\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:19.468171 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.468174    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvpgp\" (UniqueName: \"kubernetes.io/projected/569c311a-82a3-4519-87b0-2046e9b27692-kube-api-access-mvpgp\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:19.468171 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.468185    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9wb7\" (UniqueName: \"kubernetes.io/projected/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4-kube-api-access-b9wb7\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:19.873754 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.873719    2563 generic.go:356] "Generic (PLEG): container finished" podID="3d478d0c-13ea-4201-a5a9-1dfc34205439" containerID="da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79" exitCode=0
Jul 24 23:16:19.873954 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.873768    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"
Jul 24 23:16:19.873954 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.873809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f" event={"ID":"3d478d0c-13ea-4201-a5a9-1dfc34205439","Type":"ContainerDied","Data":"da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79"}
Jul 24 23:16:19.873954 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.873868    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f" event={"ID":"3d478d0c-13ea-4201-a5a9-1dfc34205439","Type":"ContainerDied","Data":"fc2e13059ffe6af781871e44bddbb3b03068ebda33b484413334297791d9092b"}
Jul 24 23:16:19.873954 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.873896    2563 scope.go:117] "RemoveContainer" containerID="da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79"
Jul 24 23:16:19.875334 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.875303    2563 generic.go:356] "Generic (PLEG): container finished" podID="569c311a-82a3-4519-87b0-2046e9b27692" containerID="3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0" exitCode=0
Jul 24 23:16:19.875461 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.875339    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"
Jul 24 23:16:19.875461 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.875355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm" event={"ID":"569c311a-82a3-4519-87b0-2046e9b27692","Type":"ContainerDied","Data":"3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0"}
Jul 24 23:16:19.875461 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.875408    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm" event={"ID":"569c311a-82a3-4519-87b0-2046e9b27692","Type":"ContainerDied","Data":"d1ca5dcc979cef7f5fe438fdf4ebaeaf17b8c36b3efc434bf772abd0f10f2541"}
Jul 24 23:16:19.877096 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.877068    2563 generic.go:356] "Generic (PLEG): container finished" podID="0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" containerID="2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c" exitCode=0
Jul 24 23:16:19.877208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.877121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq" event={"ID":"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4","Type":"ContainerDied","Data":"2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c"}
Jul 24 23:16:19.877208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.877173    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"
Jul 24 23:16:19.877313 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.877174    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq" event={"ID":"0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4","Type":"ContainerDied","Data":"1da1119a3478940341557eea81829702f184700068b8ff416e15cef073eb2a96"}
Jul 24 23:16:19.884479 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.884452    2563 scope.go:117] "RemoveContainer" containerID="da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79"
Jul 24 23:16:19.884828 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:16:19.884810    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79\": container with ID starting with da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79 not found: ID does not exist" containerID="da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79"
Jul 24 23:16:19.884927 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.884840    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79"} err="failed to get container status \"da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79\": rpc error: code = NotFound desc = could not find container \"da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79\": container with ID starting with da6ce291ed027a0390a777ac397c840b6a451e7b9191e6058f6ba33455deeb79 not found: ID does not exist"
Jul 24 23:16:19.884927 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.884862    2563 scope.go:117] "RemoveContainer" containerID="3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0"
Jul 24 23:16:19.893242 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.893220    2563 scope.go:117] "RemoveContainer" containerID="3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0"
Jul 24 23:16:19.893583 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:16:19.893565    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0\": container with ID starting with 3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0 not found: ID does not exist" containerID="3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0"
Jul 24 23:16:19.893635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.893594    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0"} err="failed to get container status \"3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0\": rpc error: code = NotFound desc = could not find container \"3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0\": container with ID starting with 3adfc503ab9a51976439bf79647e71b2de1e1147cf017451fa4b04e964ea79c0 not found: ID does not exist"
Jul 24 23:16:19.893635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.893613    2563 scope.go:117] "RemoveContainer" containerID="2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c"
Jul 24 23:16:19.898249 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.898222    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"]
Jul 24 23:16:19.901142 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.901107    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f5e9b15b-6f77988557-7kj7f"]
Jul 24 23:16:19.905044 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.905012    2563 scope.go:117] "RemoveContainer" containerID="2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c"
Jul 24 23:16:19.905488 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:16:19.905453    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c\": container with ID starting with 2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c not found: ID does not exist" containerID="2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c"
Jul 24 23:16:19.905612 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.905511    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c"} err="failed to get container status \"2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c\": rpc error: code = NotFound desc = could not find container \"2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c\": container with ID starting with 2d228cdb21866ec9ea1a022f64b8ea308bfc20b57ae146c68d9d37d93893861c not found: ID does not exist"
Jul 24 23:16:19.910295 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.910263    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"]
Jul 24 23:16:19.913425 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.913395    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f5e9b15b-dd4574b6d-lrjxq"]
Jul 24 23:16:19.923486 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.923448    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"]
Jul 24 23:16:19.927149 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:19.927120    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f5e9b15b-859f8dd696-75wfm"]
Jul 24 23:16:21.830462 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:21.830376    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:16:21.841605 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:21.841572    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" path="/var/lib/kubelet/pods/0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4/volumes"
Jul 24 23:16:21.841962 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:21.841947    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d478d0c-13ea-4201-a5a9-1dfc34205439" path="/var/lib/kubelet/pods/3d478d0c-13ea-4201-a5a9-1dfc34205439/volumes"
Jul 24 23:16:21.842273 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:21.842261    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569c311a-82a3-4519-87b0-2046e9b27692" path="/var/lib/kubelet/pods/569c311a-82a3-4519-87b0-2046e9b27692/volumes"
Jul 24 23:16:23.660189 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:23.660156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:23.758280 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:23.758241    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"]
Jul 24 23:16:23.761373 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:23.761344    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f5e9b15b-rt6fq"]
Jul 24 23:16:23.830250 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:23.830193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:23.843186 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:23.843152    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c73d77e7-1f64-4262-8d49-d2a5134bc9d4" path="/var/lib/kubelet/pods/c73d77e7-1f64-4262-8d49-d2a5134bc9d4/volumes"
Jul 24 23:16:23.858904 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:23.858877    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:16:23.860881 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:23.860857    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:16:24.061586 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:24.061546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:24.293186 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:24.293079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:24.628184 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:24.628148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:25.021985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:25.021895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:26.724124 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.724079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:26.797363 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.797315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:26.798975 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.798940    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"]
Jul 24 23:16:26.799446 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799429    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569c311a-82a3-4519-87b0-2046e9b27692" containerName="payload-pre-processing"
Jul 24 23:16:26.799553 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799448    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="569c311a-82a3-4519-87b0-2046e9b27692" containerName="payload-pre-processing"
Jul 24 23:16:26.799553 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799467    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c73d77e7-1f64-4262-8d49-d2a5134bc9d4" containerName="revoke-keys"
Jul 24 23:16:26.799553 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799477    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c73d77e7-1f64-4262-8d49-d2a5134bc9d4" containerName="revoke-keys"
Jul 24 23:16:26.799553 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799489    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d478d0c-13ea-4201-a5a9-1dfc34205439" containerName="payload-processing"
Jul 24 23:16:26.799553 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799498    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d478d0c-13ea-4201-a5a9-1dfc34205439" containerName="payload-processing"
Jul 24 23:16:26.799553 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799513    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" containerName="maas-api"
Jul 24 23:16:26.799553 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" containerName="maas-api"
Jul 24 23:16:26.799932 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799632    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="569c311a-82a3-4519-87b0-2046e9b27692" containerName="payload-pre-processing"
Jul 24 23:16:26.799932 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799644    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a93e03c-68bf-4fd7-bfd9-2b20b310c5e4" containerName="maas-api"
Jul 24 23:16:26.799932 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799655    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d478d0c-13ea-4201-a5a9-1dfc34205439" containerName="payload-processing"
Jul 24 23:16:26.799932 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.799664    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c73d77e7-1f64-4262-8d49-d2a5134bc9d4" containerName="revoke-keys"
Jul 24 23:16:26.802853 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.802826    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.805767 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.805741    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-69a3f527-serving-cert\""
Jul 24 23:16:26.805906 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.805772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:16:26.805906 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.805788    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:16:26.807037 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.807013    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:16:26.812270 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.812213    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"]
Jul 24 23:16:26.827577 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.827533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kw45\" (UniqueName: \"kubernetes.io/projected/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-kube-api-access-8kw45\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.827797 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.827591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-maas-api-tls\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.827797 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.827706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-openshift-service-ca\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.929116 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.929076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kw45\" (UniqueName: \"kubernetes.io/projected/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-kube-api-access-8kw45\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.929335 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.929135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-maas-api-tls\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.929335 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.929196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-openshift-service-ca\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.929848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.929825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-openshift-service-ca\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.931692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.931664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-maas-api-tls\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:26.937286 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:26.937253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kw45\" (UniqueName: \"kubernetes.io/projected/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-kube-api-access-8kw45\") pod \"maas-api-e2e-ait-69a3f527-7df766c945-9k7lh\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:27.116896 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.116856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:27.129399 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.129367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:27.225579 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.225546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:27.260572 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.260535    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"]
Jul 24 23:16:27.264109 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:16:27.264067    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc06b68b1_a8a4_4672_8f50_b775dc8ee0d8.slice/crio-dac9438873bd38afdeb3ec91783b6e2593755dd9823a34cdf76e20db9ca64b3d WatchSource:0}: Error finding container dac9438873bd38afdeb3ec91783b6e2593755dd9823a34cdf76e20db9ca64b3d: Status 404 returned error can't find the container with id dac9438873bd38afdeb3ec91783b6e2593755dd9823a34cdf76e20db9ca64b3d
Jul 24 23:16:27.404277 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.404194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:27.494079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.494011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:16:27.535015 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.534970    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"]
Jul 24 23:16:27.539683 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.539652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:27.542535 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.542496    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-69a3f527-dockercfg-z8548\""
Jul 24 23:16:27.542704 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.542497    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-69a3f527\""
Jul 24 23:16:27.548775 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.548747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"]
Jul 24 23:16:27.574221 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.574183    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"]
Jul 24 23:16:27.577563 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.577511    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:27.587431 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.587390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"]
Jul 24 23:16:27.637409 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.637358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc661084-e3d5-4851-ae88-b1c085793e6b-plugins-config-volume\") pod \"payload-processing-e2e-ait-69a3f527-766d566d94-t865s\" (UID: \"bc661084-e3d5-4851-ae88-b1c085793e6b\") " pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:27.637409 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.637409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmsqk\" (UniqueName: \"kubernetes.io/projected/f2731542-3f3a-4781-8534-064eae9dadee-kube-api-access-xmsqk\") pod \"payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6\" (UID: \"f2731542-3f3a-4781-8534-064eae9dadee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:27.637688 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.637501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2731542-3f3a-4781-8534-064eae9dadee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6\" (UID: \"f2731542-3f3a-4781-8534-064eae9dadee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:27.637688 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.637598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhbf\" (UniqueName: \"kubernetes.io/projected/bc661084-e3d5-4851-ae88-b1c085793e6b-kube-api-access-7bhbf\") pod \"payload-processing-e2e-ait-69a3f527-766d566d94-t865s\" (UID: \"bc661084-e3d5-4851-ae88-b1c085793e6b\") " pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:27.738873 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.738838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc661084-e3d5-4851-ae88-b1c085793e6b-plugins-config-volume\") pod \"payload-processing-e2e-ait-69a3f527-766d566d94-t865s\" (UID: \"bc661084-e3d5-4851-ae88-b1c085793e6b\") " pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:27.739333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.738892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmsqk\" (UniqueName: \"kubernetes.io/projected/f2731542-3f3a-4781-8534-064eae9dadee-kube-api-access-xmsqk\") pod \"payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6\" (UID: \"f2731542-3f3a-4781-8534-064eae9dadee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:27.739333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.738970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2731542-3f3a-4781-8534-064eae9dadee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6\" (UID: \"f2731542-3f3a-4781-8534-064eae9dadee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:27.739333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.739036    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhbf\" (UniqueName: \"kubernetes.io/projected/bc661084-e3d5-4851-ae88-b1c085793e6b-kube-api-access-7bhbf\") pod \"payload-processing-e2e-ait-69a3f527-766d566d94-t865s\" (UID: \"bc661084-e3d5-4851-ae88-b1c085793e6b\") " pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:27.739645 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.739628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2731542-3f3a-4781-8534-064eae9dadee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6\" (UID: \"f2731542-3f3a-4781-8534-064eae9dadee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:27.739757 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.739736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc661084-e3d5-4851-ae88-b1c085793e6b-plugins-config-volume\") pod \"payload-processing-e2e-ait-69a3f527-766d566d94-t865s\" (UID: \"bc661084-e3d5-4851-ae88-b1c085793e6b\") " pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:27.747092 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.747066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmsqk\" (UniqueName: \"kubernetes.io/projected/f2731542-3f3a-4781-8534-064eae9dadee-kube-api-access-xmsqk\") pod \"payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6\" (UID: \"f2731542-3f3a-4781-8534-064eae9dadee\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:27.747239 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.747064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhbf\" (UniqueName: \"kubernetes.io/projected/bc661084-e3d5-4851-ae88-b1c085793e6b-kube-api-access-7bhbf\") pod \"payload-processing-e2e-ait-69a3f527-766d566d94-t865s\" (UID: \"bc661084-e3d5-4851-ae88-b1c085793e6b\") " pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:27.852633 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.852598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:27.888828 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.888782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:27.925372 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.925239    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh" event={"ID":"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8","Type":"ContainerStarted","Data":"479254223bbd34563eb60443f0e4b418aa0c3df5292a44ced5b6fa6a91b74a9f"}
Jul 24 23:16:27.925372 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.925311    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh" event={"ID":"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8","Type":"ContainerStarted","Data":"dac9438873bd38afdeb3ec91783b6e2593755dd9823a34cdf76e20db9ca64b3d"}
Jul 24 23:16:27.925372 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.925341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:27.946306 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:27.946238    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh" podStartSLOduration=1.4866122640000001 podStartE2EDuration="1.94621326s" totalImagesPullingTime="459.600996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:16:27.944010362 +0000 UTC m=+1804.577289303" watchObservedRunningTime="2026-07-24 23:16:27.94621326 +0000 UTC m=+1804.579492199"
Jul 24 23:16:28.009841 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:16:28.009798    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc661084_e3d5_4851_ae88_b1c085793e6b.slice/crio-8f96cdd4e0b1b3e55e96a6b58b62fdd866d5f713e62a885f90299f295fed2ac3 WatchSource:0}: Error finding container 8f96cdd4e0b1b3e55e96a6b58b62fdd866d5f713e62a885f90299f295fed2ac3: Status 404 returned error can't find the container with id 8f96cdd4e0b1b3e55e96a6b58b62fdd866d5f713e62a885f90299f295fed2ac3
Jul 24 23:16:28.013787 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.013757    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"]
Jul 24 23:16:28.054532 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.054488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"]
Jul 24 23:16:28.057116 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:16:28.057085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2731542_3f3a_4781_8534_064eae9dadee.slice/crio-a35305971c126f071bc88dc3e00d381a24610f6fd7737664b65c764c50784e7b WatchSource:0}: Error finding container a35305971c126f071bc88dc3e00d381a24610f6fd7737664b65c764c50784e7b: Status 404 returned error can't find the container with id a35305971c126f071bc88dc3e00d381a24610f6fd7737664b65c764c50784e7b
Jul 24 23:16:28.935434 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.935384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6" event={"ID":"f2731542-3f3a-4781-8534-064eae9dadee","Type":"ContainerStarted","Data":"6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf"}
Jul 24 23:16:28.935938 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.935449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6" event={"ID":"f2731542-3f3a-4781-8534-064eae9dadee","Type":"ContainerStarted","Data":"a35305971c126f071bc88dc3e00d381a24610f6fd7737664b65c764c50784e7b"}
Jul 24 23:16:28.935938 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.935481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:28.937366 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.937336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s" event={"ID":"bc661084-e3d5-4851-ae88-b1c085793e6b","Type":"ContainerStarted","Data":"d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5"}
Jul 24 23:16:28.937512 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.937389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s" event={"ID":"bc661084-e3d5-4851-ae88-b1c085793e6b","Type":"ContainerStarted","Data":"8f96cdd4e0b1b3e55e96a6b58b62fdd866d5f713e62a885f90299f295fed2ac3"}
Jul 24 23:16:28.937512 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.937443    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:28.951770 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.951723    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6" podStartSLOduration=1.951706931 podStartE2EDuration="1.951706931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:16:28.950162707 +0000 UTC m=+1805.583441648" watchObservedRunningTime="2026-07-24 23:16:28.951706931 +0000 UTC m=+1805.584985872"
Jul 24 23:16:28.966782 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:28.966725    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s" podStartSLOduration=1.966703243 podStartE2EDuration="1.966703243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:16:28.965272989 +0000 UTC m=+1805.598551930" watchObservedRunningTime="2026-07-24 23:16:28.966703243 +0000 UTC m=+1805.599982218"
Jul 24 23:16:32.924986 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:32.924947    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:16:33.943804 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:33.943771    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:16:39.944701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:39.944667    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:16:39.945184 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:39.944741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:16:52.494048 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.494006    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"]
Jul 24 23:16:52.498707 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.498683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"
Jul 24 23:16:52.501316 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.501288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:16:52.503714 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.503598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"]
Jul 24 23:16:52.667817 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.667775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b66ec71-ece4-45d7-be73-ede0496a8ab1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw\" (UID: \"3b66ec71-ece4-45d7-be73-ede0496a8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"
Jul 24 23:16:52.768718 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.768635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b66ec71-ece4-45d7-be73-ede0496a8ab1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw\" (UID: \"3b66ec71-ece4-45d7-be73-ede0496a8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"
Jul 24 23:16:52.769270 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.769250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b66ec71-ece4-45d7-be73-ede0496a8ab1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw\" (UID: \"3b66ec71-ece4-45d7-be73-ede0496a8ab1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"
Jul 24 23:16:52.811715 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.811683    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"
Jul 24 23:16:52.946348 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:52.946320    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"]
Jul 24 23:16:52.948331 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:16:52.948301    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b66ec71_ece4_45d7_be73_ede0496a8ab1.slice/crio-08f4004415a48275f6bf3b6b4c47d7b334f3070f0912a706f9127aa58cfd4579 WatchSource:0}: Error finding container 08f4004415a48275f6bf3b6b4c47d7b334f3070f0912a706f9127aa58cfd4579: Status 404 returned error can't find the container with id 08f4004415a48275f6bf3b6b4c47d7b334f3070f0912a706f9127aa58cfd4579
Jul 24 23:16:53.054948 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:53.054860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw" event={"ID":"3b66ec71-ece4-45d7-be73-ede0496a8ab1","Type":"ContainerStarted","Data":"08f4004415a48275f6bf3b6b4c47d7b334f3070f0912a706f9127aa58cfd4579"}
Jul 24 23:16:54.061258 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:54.061162    2563 generic.go:356] "Generic (PLEG): container finished" podID="3b66ec71-ece4-45d7-be73-ede0496a8ab1" containerID="db44825c442ecacb28d1d67af1e669f1c5478806bb7046917e814983580f6379" exitCode=0
Jul 24 23:16:54.061669 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:54.061251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw" event={"ID":"3b66ec71-ece4-45d7-be73-ede0496a8ab1","Type":"ContainerDied","Data":"db44825c442ecacb28d1d67af1e669f1c5478806bb7046917e814983580f6379"}
Jul 24 23:16:55.214438 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:55.214410    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"
Jul 24 23:16:55.393702 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:55.393666    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b66ec71-ece4-45d7-be73-ede0496a8ab1-maas-api-service-ca\") pod \"3b66ec71-ece4-45d7-be73-ede0496a8ab1\" (UID: \"3b66ec71-ece4-45d7-be73-ede0496a8ab1\") "
Jul 24 23:16:55.394054 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:55.394029    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b66ec71-ece4-45d7-be73-ede0496a8ab1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b66ec71-ece4-45d7-be73-ede0496a8ab1" (UID: "3b66ec71-ece4-45d7-be73-ede0496a8ab1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:16:55.495240 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:55.495205    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b66ec71-ece4-45d7-be73-ede0496a8ab1-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:16:56.071218 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:56.071185    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"
Jul 24 23:16:56.071218 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:56.071196    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw" event={"ID":"3b66ec71-ece4-45d7-be73-ede0496a8ab1","Type":"ContainerDied","Data":"08f4004415a48275f6bf3b6b4c47d7b334f3070f0912a706f9127aa58cfd4579"}
Jul 24 23:16:56.071432 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:16:56.071245    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08f4004415a48275f6bf3b6b4c47d7b334f3070f0912a706f9127aa58cfd4579"
Jul 24 23:17:02.958230 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:02.958194    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"]
Jul 24 23:17:02.958780 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:02.958431    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh" podUID="c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" containerName="maas-api" containerID="cri-o://479254223bbd34563eb60443f0e4b418aa0c3df5292a44ced5b6fa6a91b74a9f" gracePeriod=30
Jul 24 23:17:03.006987 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.006947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:03.088427 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.088393    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"]
Jul 24 23:17:03.088670 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.088643    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s" podUID="bc661084-e3d5-4851-ae88-b1c085793e6b" containerName="payload-processing" containerID="cri-o://d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5" gracePeriod=30
Jul 24 23:17:03.098922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.098888    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"]
Jul 24 23:17:03.099244 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.099217    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6" podUID="f2731542-3f3a-4781-8534-064eae9dadee" containerName="payload-pre-processing" containerID="cri-o://6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf" gracePeriod=30
Jul 24 23:17:03.106621 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.106592    2563 generic.go:356] "Generic (PLEG): container finished" podID="c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" containerID="479254223bbd34563eb60443f0e4b418aa0c3df5292a44ced5b6fa6a91b74a9f" exitCode=0
Jul 24 23:17:03.106736 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.106664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh" event={"ID":"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8","Type":"ContainerDied","Data":"479254223bbd34563eb60443f0e4b418aa0c3df5292a44ced5b6fa6a91b74a9f"}
Jul 24 23:17:03.141453 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.140986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:03.226400 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.226352    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:17:03.240115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.240085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:03.366149 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.366118    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:17:03.367736 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.367713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-openshift-service-ca\") pod \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") "
Jul 24 23:17:03.367807 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.367753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-maas-api-tls\") pod \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") "
Jul 24 23:17:03.367807 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.367793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kw45\" (UniqueName: \"kubernetes.io/projected/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-kube-api-access-8kw45\") pod \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\" (UID: \"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8\") "
Jul 24 23:17:03.368086 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.368060    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" (UID: "c06b68b1-a8a4-4672-8f50-b775dc8ee0d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:17:03.369910 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.369882    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" (UID: "c06b68b1-a8a4-4672-8f50-b775dc8ee0d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:17:03.370001 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.369941    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-kube-api-access-8kw45" (OuterVolumeSpecName: "kube-api-access-8kw45") pod "c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" (UID: "c06b68b1-a8a4-4672-8f50-b775dc8ee0d8"). InnerVolumeSpecName "kube-api-access-8kw45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:17:03.375799 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.375775    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:17:03.469022 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.468925    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bhbf\" (UniqueName: \"kubernetes.io/projected/bc661084-e3d5-4851-ae88-b1c085793e6b-kube-api-access-7bhbf\") pod \"bc661084-e3d5-4851-ae88-b1c085793e6b\" (UID: \"bc661084-e3d5-4851-ae88-b1c085793e6b\") "
Jul 24 23:17:03.469022 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.468969    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmsqk\" (UniqueName: \"kubernetes.io/projected/f2731542-3f3a-4781-8534-064eae9dadee-kube-api-access-xmsqk\") pod \"f2731542-3f3a-4781-8534-064eae9dadee\" (UID: \"f2731542-3f3a-4781-8534-064eae9dadee\") "
Jul 24 23:17:03.469022 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.468985    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2731542-3f3a-4781-8534-064eae9dadee-plugins-config-volume\") pod \"f2731542-3f3a-4781-8534-064eae9dadee\" (UID: \"f2731542-3f3a-4781-8534-064eae9dadee\") "
Jul 24 23:17:03.469248 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.469049    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc661084-e3d5-4851-ae88-b1c085793e6b-plugins-config-volume\") pod \"bc661084-e3d5-4851-ae88-b1c085793e6b\" (UID: \"bc661084-e3d5-4851-ae88-b1c085793e6b\") "
Jul 24 23:17:03.469306 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.469277    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:03.469349 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.469303    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:03.469349 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.469318    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kw45\" (UniqueName: \"kubernetes.io/projected/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8-kube-api-access-8kw45\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:03.469420 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.469400    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2731542-3f3a-4781-8534-064eae9dadee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2731542-3f3a-4781-8534-064eae9dadee" (UID: "f2731542-3f3a-4781-8534-064eae9dadee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:17:03.469478 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.469455    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc661084-e3d5-4851-ae88-b1c085793e6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc661084-e3d5-4851-ae88-b1c085793e6b" (UID: "bc661084-e3d5-4851-ae88-b1c085793e6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:17:03.471074 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.471052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc661084-e3d5-4851-ae88-b1c085793e6b-kube-api-access-7bhbf" (OuterVolumeSpecName: "kube-api-access-7bhbf") pod "bc661084-e3d5-4851-ae88-b1c085793e6b" (UID: "bc661084-e3d5-4851-ae88-b1c085793e6b"). InnerVolumeSpecName "kube-api-access-7bhbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:17:03.471182 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.471163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2731542-3f3a-4781-8534-064eae9dadee-kube-api-access-xmsqk" (OuterVolumeSpecName: "kube-api-access-xmsqk") pod "f2731542-3f3a-4781-8534-064eae9dadee" (UID: "f2731542-3f3a-4781-8534-064eae9dadee"). InnerVolumeSpecName "kube-api-access-xmsqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:17:03.570380 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.570341    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc661084-e3d5-4851-ae88-b1c085793e6b-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:03.570380 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.570372    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhbf\" (UniqueName: \"kubernetes.io/projected/bc661084-e3d5-4851-ae88-b1c085793e6b-kube-api-access-7bhbf\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:03.570380 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.570383    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmsqk\" (UniqueName: \"kubernetes.io/projected/f2731542-3f3a-4781-8534-064eae9dadee-kube-api-access-xmsqk\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:03.570675 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:03.570392    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2731542-3f3a-4781-8534-064eae9dadee-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:04.112745 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.112656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"
Jul 24 23:17:04.112745 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.112670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh" event={"ID":"c06b68b1-a8a4-4672-8f50-b775dc8ee0d8","Type":"ContainerDied","Data":"dac9438873bd38afdeb3ec91783b6e2593755dd9823a34cdf76e20db9ca64b3d"}
Jul 24 23:17:04.112745 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.112730    2563 scope.go:117] "RemoveContainer" containerID="479254223bbd34563eb60443f0e4b418aa0c3df5292a44ced5b6fa6a91b74a9f"
Jul 24 23:17:04.114237 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.114121    2563 generic.go:356] "Generic (PLEG): container finished" podID="bc661084-e3d5-4851-ae88-b1c085793e6b" containerID="d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5" exitCode=0
Jul 24 23:17:04.114237 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.114162    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"
Jul 24 23:17:04.114237 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.114201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s" event={"ID":"bc661084-e3d5-4851-ae88-b1c085793e6b","Type":"ContainerDied","Data":"d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5"}
Jul 24 23:17:04.114407 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.114259    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s" event={"ID":"bc661084-e3d5-4851-ae88-b1c085793e6b","Type":"ContainerDied","Data":"8f96cdd4e0b1b3e55e96a6b58b62fdd866d5f713e62a885f90299f295fed2ac3"}
Jul 24 23:17:04.115832 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.115810    2563 generic.go:356] "Generic (PLEG): container finished" podID="f2731542-3f3a-4781-8534-064eae9dadee" containerID="6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf" exitCode=0
Jul 24 23:17:04.115963 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.115856    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"
Jul 24 23:17:04.115963 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.115873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6" event={"ID":"f2731542-3f3a-4781-8534-064eae9dadee","Type":"ContainerDied","Data":"6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf"}
Jul 24 23:17:04.115963 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.115910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6" event={"ID":"f2731542-3f3a-4781-8534-064eae9dadee","Type":"ContainerDied","Data":"a35305971c126f071bc88dc3e00d381a24610f6fd7737664b65c764c50784e7b"}
Jul 24 23:17:04.122756 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.122726    2563 scope.go:117] "RemoveContainer" containerID="d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5"
Jul 24 23:17:04.131274 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.131252    2563 scope.go:117] "RemoveContainer" containerID="d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5"
Jul 24 23:17:04.131618 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:17:04.131593    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5\": container with ID starting with d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5 not found: ID does not exist" containerID="d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5"
Jul 24 23:17:04.131708 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.131626    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5"} err="failed to get container status \"d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5\": rpc error: code = NotFound desc = could not find container \"d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5\": container with ID starting with d52014b119619585f2817900ce95432bc98d776126a793c10ba8a131651f52e5 not found: ID does not exist"
Jul 24 23:17:04.131708 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.131643    2563 scope.go:117] "RemoveContainer" containerID="6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf"
Jul 24 23:17:04.135581 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.135549    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"]
Jul 24 23:17:04.139435 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.139408    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-69a3f527-766d566d94-t865s"]
Jul 24 23:17:04.140442 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.140423    2563 scope.go:117] "RemoveContainer" containerID="6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf"
Jul 24 23:17:04.140775 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:17:04.140756    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf\": container with ID starting with 6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf not found: ID does not exist" containerID="6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf"
Jul 24 23:17:04.140848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.140780    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf"} err="failed to get container status \"6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf\": rpc error: code = NotFound desc = could not find container \"6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf\": container with ID starting with 6d5097c4d29b056437f939afb448536230ad39998544402ed6c73241f3011fdf not found: ID does not exist"
Jul 24 23:17:04.148987 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.148957    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"]
Jul 24 23:17:04.154784 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.154757    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-69a3f527-7df766c945-9k7lh"]
Jul 24 23:17:04.165335 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.165299    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"]
Jul 24 23:17:04.168772 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:04.168740    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-69a3f527-86797cdd8d-s45r6"]
Jul 24 23:17:05.842389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:05.842349    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc661084-e3d5-4851-ae88-b1c085793e6b" path="/var/lib/kubelet/pods/bc661084-e3d5-4851-ae88-b1c085793e6b/volumes"
Jul 24 23:17:05.842969 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:05.842947    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" path="/var/lib/kubelet/pods/c06b68b1-a8a4-4672-8f50-b775dc8ee0d8/volumes"
Jul 24 23:17:05.843325 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:05.843306    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2731542-3f3a-4781-8534-064eae9dadee" path="/var/lib/kubelet/pods/f2731542-3f3a-4781-8534-064eae9dadee/volumes"
Jul 24 23:17:06.256134 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:06.256037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:17:07.570975 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:07.570941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:07.575912 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:07.575880    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"]
Jul 24 23:17:07.578008 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:07.577980    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-69a3f527-rjkfw"]
Jul 24 23:17:07.842046 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:07.842016    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b66ec71-ece4-45d7-be73-ede0496a8ab1" path="/var/lib/kubelet/pods/3b66ec71-ece4-45d7-be73-ede0496a8ab1/volumes"
Jul 24 23:17:08.787688 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:08.787659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:09.019199 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:09.019165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:09.240453 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:09.240423    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:09.542115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:09.542024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:10.065296 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:10.065259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:11.969862 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:11.969823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:12.068801 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.068762    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:12.203335 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203298    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"]
Jul 24 23:17:12.203735 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203708    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b66ec71-ece4-45d7-be73-ede0496a8ab1" containerName="revoke-keys"
Jul 24 23:17:12.203735 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203737    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b66ec71-ece4-45d7-be73-ede0496a8ab1" containerName="revoke-keys"
Jul 24 23:17:12.203847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203754    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc661084-e3d5-4851-ae88-b1c085793e6b" containerName="payload-processing"
Jul 24 23:17:12.203847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203762    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc661084-e3d5-4851-ae88-b1c085793e6b" containerName="payload-processing"
Jul 24 23:17:12.203847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203771    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2731542-3f3a-4781-8534-064eae9dadee" containerName="payload-pre-processing"
Jul 24 23:17:12.203847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203776    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2731542-3f3a-4781-8534-064eae9dadee" containerName="payload-pre-processing"
Jul 24 23:17:12.203847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203786    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" containerName="maas-api"
Jul 24 23:17:12.203847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203792    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" containerName="maas-api"
Jul 24 23:17:12.203847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203841    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2731542-3f3a-4781-8534-064eae9dadee" containerName="payload-pre-processing"
Jul 24 23:17:12.203847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203851    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c06b68b1-a8a4-4672-8f50-b775dc8ee0d8" containerName="maas-api"
Jul 24 23:17:12.204115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203857    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b66ec71-ece4-45d7-be73-ede0496a8ab1" containerName="revoke-keys"
Jul 24 23:17:12.204115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.203866    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc661084-e3d5-4851-ae88-b1c085793e6b" containerName="payload-processing"
Jul 24 23:17:12.208425 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.208399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.211048 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.211021    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:17:12.211310 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.211293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:17:12.212284 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.212265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-57bcbd39-serving-cert\""
Jul 24 23:17:12.212403 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.212265    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:17:12.217966 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.217932    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"]
Jul 24 23:17:12.240445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.240351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:12.242541 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.242498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-openshift-service-ca\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.242630 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.242603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptgv9\" (UniqueName: \"kubernetes.io/projected/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-kube-api-access-ptgv9\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.242669 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.242645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-maas-api-tls\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.317983 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.317948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:12.343317 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.343283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-openshift-service-ca\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.343479 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.343348    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptgv9\" (UniqueName: \"kubernetes.io/projected/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-kube-api-access-ptgv9\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.343479 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.343384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-maas-api-tls\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.344013 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.343986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-openshift-service-ca\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.345899 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.345872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-maas-api-tls\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.352167 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.352134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptgv9\" (UniqueName: \"kubernetes.io/projected/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-kube-api-access-ptgv9\") pod \"maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.522714 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.522617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:12.701489 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.701446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"]
Jul 24 23:17:12.704860 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:17:12.704819    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269bb9fe_ac7a_4ecd_9f00_aeed36ae9361.slice/crio-0579356ee96975ef9f8cd71e87bd0714ce5908e1889b3289cda002c658fc4d99 WatchSource:0}: Error finding container 0579356ee96975ef9f8cd71e87bd0714ce5908e1889b3289cda002c658fc4d99: Status 404 returned error can't find the container with id 0579356ee96975ef9f8cd71e87bd0714ce5908e1889b3289cda002c658fc4d99
Jul 24 23:17:12.821708 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.821605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:12.924773 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.924735    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"]
Jul 24 23:17:12.929096 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.929072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:12.930109 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.930084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:12.931834 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.931816    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-57bcbd39\""
Jul 24 23:17:12.931911 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.931816    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-57bcbd39-dockercfg-z78q6\""
Jul 24 23:17:12.939191 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.939158    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"]
Jul 24 23:17:12.949890 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.949847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj8q5\" (UniqueName: \"kubernetes.io/projected/24706302-d913-4780-8c45-6e9bbf637b94-kube-api-access-hj8q5\") pod \"payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6\" (UID: \"24706302-d913-4780-8c45-6e9bbf637b94\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:12.950072 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.949944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24706302-d913-4780-8c45-6e9bbf637b94-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6\" (UID: \"24706302-d913-4780-8c45-6e9bbf637b94\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:12.961536 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.961486    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"]
Jul 24 23:17:12.964784 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.964762    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:12.974982 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:12.974951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"]
Jul 24 23:17:13.050486 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.050440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1766f0cf-c81d-4791-8f72-15e1ec403212-plugins-config-volume\") pod \"payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d\" (UID: \"1766f0cf-c81d-4791-8f72-15e1ec403212\") " pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:13.050736 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.050584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj8q5\" (UniqueName: \"kubernetes.io/projected/24706302-d913-4780-8c45-6e9bbf637b94-kube-api-access-hj8q5\") pod \"payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6\" (UID: \"24706302-d913-4780-8c45-6e9bbf637b94\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:13.050736 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.050631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24706302-d913-4780-8c45-6e9bbf637b94-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6\" (UID: \"24706302-d913-4780-8c45-6e9bbf637b94\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:13.050736 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.050671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfmq\" (UniqueName: \"kubernetes.io/projected/1766f0cf-c81d-4791-8f72-15e1ec403212-kube-api-access-rjfmq\") pod \"payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d\" (UID: \"1766f0cf-c81d-4791-8f72-15e1ec403212\") " pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:13.051393 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.051366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24706302-d913-4780-8c45-6e9bbf637b94-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6\" (UID: \"24706302-d913-4780-8c45-6e9bbf637b94\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:13.058609 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.058580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj8q5\" (UniqueName: \"kubernetes.io/projected/24706302-d913-4780-8c45-6e9bbf637b94-kube-api-access-hj8q5\") pod \"payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6\" (UID: \"24706302-d913-4780-8c45-6e9bbf637b94\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:13.151627 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.151590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1766f0cf-c81d-4791-8f72-15e1ec403212-plugins-config-volume\") pod \"payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d\" (UID: \"1766f0cf-c81d-4791-8f72-15e1ec403212\") " pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:13.151852 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.151701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfmq\" (UniqueName: \"kubernetes.io/projected/1766f0cf-c81d-4791-8f72-15e1ec403212-kube-api-access-rjfmq\") pod \"payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d\" (UID: \"1766f0cf-c81d-4791-8f72-15e1ec403212\") " pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:13.152397 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.152375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1766f0cf-c81d-4791-8f72-15e1ec403212-plugins-config-volume\") pod \"payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d\" (UID: \"1766f0cf-c81d-4791-8f72-15e1ec403212\") " pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:13.159502 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.159463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfmq\" (UniqueName: \"kubernetes.io/projected/1766f0cf-c81d-4791-8f72-15e1ec403212-kube-api-access-rjfmq\") pod \"payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d\" (UID: \"1766f0cf-c81d-4791-8f72-15e1ec403212\") " pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:13.167783 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.167750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr" event={"ID":"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361","Type":"ContainerStarted","Data":"0579356ee96975ef9f8cd71e87bd0714ce5908e1889b3289cda002c658fc4d99"}
Jul 24 23:17:13.241148 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.241101    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:13.276579 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.276540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:13.391040 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.391012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"]
Jul 24 23:17:13.393767 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:17:13.393723    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24706302_d913_4780_8c45_6e9bbf637b94.slice/crio-cddead31fb2341bcd360afd227155917e342d7454896bb50615171704a212226 WatchSource:0}: Error finding container cddead31fb2341bcd360afd227155917e342d7454896bb50615171704a212226: Status 404 returned error can't find the container with id cddead31fb2341bcd360afd227155917e342d7454896bb50615171704a212226
Jul 24 23:17:13.431157 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:13.430869    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"]
Jul 24 23:17:13.433223 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:17:13.433192    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1766f0cf_c81d_4791_8f72_15e1ec403212.slice/crio-7c11ac74f27b791f65f97e9588e494d6002b516618befdf3269b0337e25107a9 WatchSource:0}: Error finding container 7c11ac74f27b791f65f97e9588e494d6002b516618befdf3269b0337e25107a9: Status 404 returned error can't find the container with id 7c11ac74f27b791f65f97e9588e494d6002b516618befdf3269b0337e25107a9
Jul 24 23:17:14.178103 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.178069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6" event={"ID":"24706302-d913-4780-8c45-6e9bbf637b94","Type":"ContainerStarted","Data":"4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44"}
Jul 24 23:17:14.178589 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.178124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6" event={"ID":"24706302-d913-4780-8c45-6e9bbf637b94","Type":"ContainerStarted","Data":"cddead31fb2341bcd360afd227155917e342d7454896bb50615171704a212226"}
Jul 24 23:17:14.178589 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.178181    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:14.179820 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.179797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr" event={"ID":"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361","Type":"ContainerStarted","Data":"81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec"}
Jul 24 23:17:14.179938 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.179848    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:14.181392 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.181364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d" event={"ID":"1766f0cf-c81d-4791-8f72-15e1ec403212","Type":"ContainerStarted","Data":"fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306"}
Jul 24 23:17:14.181501 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.181414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d" event={"ID":"1766f0cf-c81d-4791-8f72-15e1ec403212","Type":"ContainerStarted","Data":"7c11ac74f27b791f65f97e9588e494d6002b516618befdf3269b0337e25107a9"}
Jul 24 23:17:14.181501 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.181442    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:14.194985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.194932    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6" podStartSLOduration=2.194915498 podStartE2EDuration="2.194915498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:17:14.19396695 +0000 UTC m=+1850.827245892" watchObservedRunningTime="2026-07-24 23:17:14.194915498 +0000 UTC m=+1850.828194438"
Jul 24 23:17:14.210656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.210552    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr" podStartSLOduration=1.415727845 podStartE2EDuration="2.210501423s" totalImagesPullingTime="794.773578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:17:14.209878591 +0000 UTC m=+1850.843157525" watchObservedRunningTime="2026-07-24 23:17:14.210501423 +0000 UTC m=+1850.843780365"
Jul 24 23:17:14.225651 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:14.225597    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d" podStartSLOduration=2.225578148 podStartE2EDuration="2.225578148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:17:14.224594632 +0000 UTC m=+1850.857873574" watchObservedRunningTime="2026-07-24 23:17:14.225578148 +0000 UTC m=+1850.858857088"
Jul 24 23:17:17.138594 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:17.137818    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:17:20.192443 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:20.192415    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:25.189163 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:25.189129    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:25.189604 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:25.189212    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:36.850799 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:36.850756    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"]
Jul 24 23:17:36.854017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:36.853998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"
Jul 24 23:17:36.856804 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:36.856778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:17:36.861340 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:36.861284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"]
Jul 24 23:17:36.965059 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:36.965023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e254095-e82b-4df6-8e18-57b2d1af1073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv\" (UID: \"7e254095-e82b-4df6-8e18-57b2d1af1073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"
Jul 24 23:17:37.066007 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:37.065971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e254095-e82b-4df6-8e18-57b2d1af1073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv\" (UID: \"7e254095-e82b-4df6-8e18-57b2d1af1073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"
Jul 24 23:17:37.066665 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:37.066644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e254095-e82b-4df6-8e18-57b2d1af1073-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv\" (UID: \"7e254095-e82b-4df6-8e18-57b2d1af1073\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"
Jul 24 23:17:37.166243 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:37.166159    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"
Jul 24 23:17:37.306624 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:37.306587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"]
Jul 24 23:17:37.310808 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:17:37.310775    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e254095_e82b_4df6_8e18_57b2d1af1073.slice/crio-043cf9794e14081bd1bcbc35bea3851d2c7ca3029f6c6c568b461b2b5be986d8 WatchSource:0}: Error finding container 043cf9794e14081bd1bcbc35bea3851d2c7ca3029f6c6c568b461b2b5be986d8: Status 404 returned error can't find the container with id 043cf9794e14081bd1bcbc35bea3851d2c7ca3029f6c6c568b461b2b5be986d8
Jul 24 23:17:38.312873 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:38.312773    2563 generic.go:356] "Generic (PLEG): container finished" podID="7e254095-e82b-4df6-8e18-57b2d1af1073" containerID="6d2b2143b642d25e3afe7e130c0dc4bef6514f36e5f88a4ed1d1935f094554f6" exitCode=0
Jul 24 23:17:38.313238 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:38.312860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv" event={"ID":"7e254095-e82b-4df6-8e18-57b2d1af1073","Type":"ContainerDied","Data":"6d2b2143b642d25e3afe7e130c0dc4bef6514f36e5f88a4ed1d1935f094554f6"}
Jul 24 23:17:38.313238 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:38.312913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv" event={"ID":"7e254095-e82b-4df6-8e18-57b2d1af1073","Type":"ContainerStarted","Data":"043cf9794e14081bd1bcbc35bea3851d2c7ca3029f6c6c568b461b2b5be986d8"}
Jul 24 23:17:39.457071 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:39.457040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"
Jul 24 23:17:39.590715 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:39.590620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e254095-e82b-4df6-8e18-57b2d1af1073-maas-api-service-ca\") pod \"7e254095-e82b-4df6-8e18-57b2d1af1073\" (UID: \"7e254095-e82b-4df6-8e18-57b2d1af1073\") "
Jul 24 23:17:39.591082 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:39.591056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e254095-e82b-4df6-8e18-57b2d1af1073-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e254095-e82b-4df6-8e18-57b2d1af1073" (UID: "7e254095-e82b-4df6-8e18-57b2d1af1073"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:17:39.692007 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:39.691963    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e254095-e82b-4df6-8e18-57b2d1af1073-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:40.324005 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:40.323923    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"
Jul 24 23:17:40.324005 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:40.323971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv" event={"ID":"7e254095-e82b-4df6-8e18-57b2d1af1073","Type":"ContainerDied","Data":"043cf9794e14081bd1bcbc35bea3851d2c7ca3029f6c6c568b461b2b5be986d8"}
Jul 24 23:17:40.324192 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:40.324017    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="043cf9794e14081bd1bcbc35bea3851d2c7ca3029f6c6c568b461b2b5be986d8"
Jul 24 23:17:47.406128 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.406027    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"]
Jul 24 23:17:47.406654 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.406340    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr" podUID="269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" containerName="maas-api" containerID="cri-o://81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec" gracePeriod=30
Jul 24 23:17:47.457096 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.457060    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"]
Jul 24 23:17:47.457711 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.457328    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d" podUID="1766f0cf-c81d-4791-8f72-15e1ec403212" containerName="payload-processing" containerID="cri-o://fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306" gracePeriod=30
Jul 24 23:17:47.465750 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.465714    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"]
Jul 24 23:17:47.466158 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.466121    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6" podUID="24706302-d913-4780-8c45-6e9bbf637b94" containerName="payload-pre-processing" containerID="cri-o://4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44" gracePeriod=30
Jul 24 23:17:47.481065 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.481035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:47.580941 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.580903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:47.747814 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.747784    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:47.760289 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.760268    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:47.766116 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.766090    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:47.866018 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.865981    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptgv9\" (UniqueName: \"kubernetes.io/projected/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-kube-api-access-ptgv9\") pod \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") "
Jul 24 23:17:47.866229 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866025    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24706302-d913-4780-8c45-6e9bbf637b94-plugins-config-volume\") pod \"24706302-d913-4780-8c45-6e9bbf637b94\" (UID: \"24706302-d913-4780-8c45-6e9bbf637b94\") "
Jul 24 23:17:47.866229 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-maas-api-tls\") pod \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") "
Jul 24 23:17:47.866229 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866112    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjfmq\" (UniqueName: \"kubernetes.io/projected/1766f0cf-c81d-4791-8f72-15e1ec403212-kube-api-access-rjfmq\") pod \"1766f0cf-c81d-4791-8f72-15e1ec403212\" (UID: \"1766f0cf-c81d-4791-8f72-15e1ec403212\") "
Jul 24 23:17:47.866229 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1766f0cf-c81d-4791-8f72-15e1ec403212-plugins-config-volume\") pod \"1766f0cf-c81d-4791-8f72-15e1ec403212\" (UID: \"1766f0cf-c81d-4791-8f72-15e1ec403212\") "
Jul 24 23:17:47.866445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj8q5\" (UniqueName: \"kubernetes.io/projected/24706302-d913-4780-8c45-6e9bbf637b94-kube-api-access-hj8q5\") pod \"24706302-d913-4780-8c45-6e9bbf637b94\" (UID: \"24706302-d913-4780-8c45-6e9bbf637b94\") "
Jul 24 23:17:47.866445 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866387    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-openshift-service-ca\") pod \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\" (UID: \"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361\") "
Jul 24 23:17:47.866641 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866606    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24706302-d913-4780-8c45-6e9bbf637b94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24706302-d913-4780-8c45-6e9bbf637b94" (UID: "24706302-d913-4780-8c45-6e9bbf637b94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:17:47.866721 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866700    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24706302-d913-4780-8c45-6e9bbf637b94-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:47.866816 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866688    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1766f0cf-c81d-4791-8f72-15e1ec403212-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1766f0cf-c81d-4791-8f72-15e1ec403212" (UID: "1766f0cf-c81d-4791-8f72-15e1ec403212"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:17:47.866879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.866844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" (UID: "269bb9fe-ac7a-4ecd-9f00-aeed36ae9361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:17:47.868425 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.868395    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" (UID: "269bb9fe-ac7a-4ecd-9f00-aeed36ae9361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:17:47.868556 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.868425    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24706302-d913-4780-8c45-6e9bbf637b94-kube-api-access-hj8q5" (OuterVolumeSpecName: "kube-api-access-hj8q5") pod "24706302-d913-4780-8c45-6e9bbf637b94" (UID: "24706302-d913-4780-8c45-6e9bbf637b94"). InnerVolumeSpecName "kube-api-access-hj8q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:17:47.868556 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.868454    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1766f0cf-c81d-4791-8f72-15e1ec403212-kube-api-access-rjfmq" (OuterVolumeSpecName: "kube-api-access-rjfmq") pod "1766f0cf-c81d-4791-8f72-15e1ec403212" (UID: "1766f0cf-c81d-4791-8f72-15e1ec403212"). InnerVolumeSpecName "kube-api-access-rjfmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:17:47.868556 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.868472    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-kube-api-access-ptgv9" (OuterVolumeSpecName: "kube-api-access-ptgv9") pod "269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" (UID: "269bb9fe-ac7a-4ecd-9f00-aeed36ae9361"). InnerVolumeSpecName "kube-api-access-ptgv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:17:47.967900 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.967794    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1766f0cf-c81d-4791-8f72-15e1ec403212-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:47.967900 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.967831    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj8q5\" (UniqueName: \"kubernetes.io/projected/24706302-d913-4780-8c45-6e9bbf637b94-kube-api-access-hj8q5\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:47.967900 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.967844    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:47.967900 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.967855    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptgv9\" (UniqueName: \"kubernetes.io/projected/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-kube-api-access-ptgv9\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:47.967900 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.967869    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:47.967900 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:47.967881    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfmq\" (UniqueName: \"kubernetes.io/projected/1766f0cf-c81d-4791-8f72-15e1ec403212-kube-api-access-rjfmq\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:17:48.361332 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.361291    2563 generic.go:356] "Generic (PLEG): container finished" podID="24706302-d913-4780-8c45-6e9bbf637b94" containerID="4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44" exitCode=0
Jul 24 23:17:48.361571 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.361362    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"
Jul 24 23:17:48.361571 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.361380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6" event={"ID":"24706302-d913-4780-8c45-6e9bbf637b94","Type":"ContainerDied","Data":"4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44"}
Jul 24 23:17:48.361571 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.361441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6" event={"ID":"24706302-d913-4780-8c45-6e9bbf637b94","Type":"ContainerDied","Data":"cddead31fb2341bcd360afd227155917e342d7454896bb50615171704a212226"}
Jul 24 23:17:48.361571 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.361469    2563 scope.go:117] "RemoveContainer" containerID="4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44"
Jul 24 23:17:48.362735 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.362712    2563 generic.go:356] "Generic (PLEG): container finished" podID="269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" containerID="81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec" exitCode=0
Jul 24 23:17:48.362850 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.362791    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"
Jul 24 23:17:48.362913 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.362794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr" event={"ID":"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361","Type":"ContainerDied","Data":"81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec"}
Jul 24 23:17:48.362971 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.362942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr" event={"ID":"269bb9fe-ac7a-4ecd-9f00-aeed36ae9361","Type":"ContainerDied","Data":"0579356ee96975ef9f8cd71e87bd0714ce5908e1889b3289cda002c658fc4d99"}
Jul 24 23:17:48.364088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.364062    2563 generic.go:356] "Generic (PLEG): container finished" podID="1766f0cf-c81d-4791-8f72-15e1ec403212" containerID="fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306" exitCode=0
Jul 24 23:17:48.364211 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.364105    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"
Jul 24 23:17:48.364211 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.364144    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d" event={"ID":"1766f0cf-c81d-4791-8f72-15e1ec403212","Type":"ContainerDied","Data":"fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306"}
Jul 24 23:17:48.364211 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.364188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d" event={"ID":"1766f0cf-c81d-4791-8f72-15e1ec403212","Type":"ContainerDied","Data":"7c11ac74f27b791f65f97e9588e494d6002b516618befdf3269b0337e25107a9"}
Jul 24 23:17:48.371908 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.371837    2563 scope.go:117] "RemoveContainer" containerID="4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44"
Jul 24 23:17:48.372302 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:17:48.372236    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44\": container with ID starting with 4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44 not found: ID does not exist" containerID="4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44"
Jul 24 23:17:48.372302 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.372276    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44"} err="failed to get container status \"4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44\": rpc error: code = NotFound desc = could not find container \"4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44\": container with ID starting with 4fffc064c862319966dc9125ccf0518eb8dd66e47096d9adab5babd9cb946a44 not found: ID does not exist"
Jul 24 23:17:48.372302 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.372302    2563 scope.go:117] "RemoveContainer" containerID="81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec"
Jul 24 23:17:48.381208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.381188    2563 scope.go:117] "RemoveContainer" containerID="81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec"
Jul 24 23:17:48.381566 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:17:48.381547    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec\": container with ID starting with 81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec not found: ID does not exist" containerID="81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec"
Jul 24 23:17:48.381624 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.381575    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec"} err="failed to get container status \"81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec\": rpc error: code = NotFound desc = could not find container \"81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec\": container with ID starting with 81efbf550dfb93a63e86466a957bf6c4e51f389060d52d81bd7a4fc351690fec not found: ID does not exist"
Jul 24 23:17:48.381624 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.381593    2563 scope.go:117] "RemoveContainer" containerID="fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306"
Jul 24 23:17:48.389748 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.389712    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"]
Jul 24 23:17:48.391737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.391453    2563 scope.go:117] "RemoveContainer" containerID="fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306"
Jul 24 23:17:48.391926 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:17:48.391899    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306\": container with ID starting with fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306 not found: ID does not exist" containerID="fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306"
Jul 24 23:17:48.391999 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.391935    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306"} err="failed to get container status \"fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306\": rpc error: code = NotFound desc = could not find container \"fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306\": container with ID starting with fd5dd171a1f8afd730f5cab53dab3fed5161c8e84936f82ba34ab8ad6244c306 not found: ID does not exist"
Jul 24 23:17:48.393465 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.393446    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-57bcbd39-6c8cb99f68-pnvf6"]
Jul 24 23:17:48.404761 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.404725    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"]
Jul 24 23:17:48.408175 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.408136    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-57bcbd39-5b6b5cbc8b-r2gcr"]
Jul 24 23:17:48.418644 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.418612    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"]
Jul 24 23:17:48.420322 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:48.420295    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-57bcbd39-65584566f8-jr52d"]
Jul 24 23:17:49.841105 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:49.841069    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1766f0cf-c81d-4791-8f72-15e1ec403212" path="/var/lib/kubelet/pods/1766f0cf-c81d-4791-8f72-15e1ec403212/volumes"
Jul 24 23:17:49.841505 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:49.841420    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24706302-d913-4780-8c45-6e9bbf637b94" path="/var/lib/kubelet/pods/24706302-d913-4780-8c45-6e9bbf637b94/volumes"
Jul 24 23:17:49.841804 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:49.841789    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" path="/var/lib/kubelet/pods/269bb9fe-ac7a-4ecd-9f00-aeed36ae9361/volumes"
Jul 24 23:17:50.225751 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:50.225660    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:17:51.923457 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:51.923411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:51.934290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:51.934254    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"]
Jul 24 23:17:51.936869 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:51.936837    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-57bcbd39-s4tsv"]
Jul 24 23:17:52.092551 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:52.092502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:52.330101 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:52.329957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:52.813263 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:52.813225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:52.897830 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:52.897790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:52.999176 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:52.999119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:17:53.841295 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:53.841254    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e254095-e82b-4df6-8e18-57b2d1af1073" path="/var/lib/kubelet/pods/7e254095-e82b-4df6-8e18-57b2d1af1073/volumes"
Jul 24 23:17:54.399370 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:17:54.399339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:00.389997 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:00.389947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:00.492646 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:00.492600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:00.591124 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:00.591084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:00.690665 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:00.690574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:01.114232 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:01.114193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:15.081234 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:15.081187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:15.184426 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:15.184394    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:27.069283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:27.069223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:33.641120 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:33.641086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:33.748294 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:33.748252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:33.847738 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:33.847684    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:49.746571 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:49.746498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:55.552917 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:55.552885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:55.725809 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:55.725755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:18:55.827992 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:18:55.827888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:06.306769 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:06.306732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:07.094742 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:07.094712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:13.542478 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:13.542443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:13.647499 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:13.647456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:13.744981 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:13.744954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:13.844504 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:13.844475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:14.639572 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:14.639530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:14.738884 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:14.738842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:14.868076 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:14.868041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:16.438140 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:16.438056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:17.047632 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:17.047591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:19:22.864209 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:22.864156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:24.003230 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:24.003198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:26.631385 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:26.631345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:19:30.100289 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:30.100238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:30.171510 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:30.171472    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:30.338168 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:30.338120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:30.793043 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:30.792995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:30.973770 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:30.973741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:33.386117 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.386081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:33.470147 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470113    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"]
Jul 24 23:19:33.470625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470608    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" containerName="maas-api"
Jul 24 23:19:33.470701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470628    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" containerName="maas-api"
Jul 24 23:19:33.470701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470651    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e254095-e82b-4df6-8e18-57b2d1af1073" containerName="revoke-keys"
Jul 24 23:19:33.470701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470659    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e254095-e82b-4df6-8e18-57b2d1af1073" containerName="revoke-keys"
Jul 24 23:19:33.470701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470678    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24706302-d913-4780-8c45-6e9bbf637b94" containerName="payload-pre-processing"
Jul 24 23:19:33.470701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470688    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="24706302-d913-4780-8c45-6e9bbf637b94" containerName="payload-pre-processing"
Jul 24 23:19:33.470882 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470707    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1766f0cf-c81d-4791-8f72-15e1ec403212" containerName="payload-processing"
Jul 24 23:19:33.470882 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1766f0cf-c81d-4791-8f72-15e1ec403212" containerName="payload-processing"
Jul 24 23:19:33.470882 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470789    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="269bb9fe-ac7a-4ecd-9f00-aeed36ae9361" containerName="maas-api"
Jul 24 23:19:33.470882 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470802    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1766f0cf-c81d-4791-8f72-15e1ec403212" containerName="payload-processing"
Jul 24 23:19:33.470882 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470815    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="24706302-d913-4780-8c45-6e9bbf637b94" containerName="payload-pre-processing"
Jul 24 23:19:33.470882 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.470825    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e254095-e82b-4df6-8e18-57b2d1af1073" containerName="revoke-keys"
Jul 24 23:19:33.474693 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.474650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.475295 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.475270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:33.477506 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.477480    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:19:33.477791 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.477770    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:19:33.478685 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.478667    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-93ae309a-serving-cert\""
Jul 24 23:19:33.478810 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.478780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:19:33.482816 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.482773    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"]
Jul 24 23:19:33.492042 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.492008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zjv\" (UniqueName: \"kubernetes.io/projected/1741c8cf-c097-428a-b458-bf3bf281df5c-kube-api-access-47zjv\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.492197 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.492067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1741c8cf-c097-428a-b458-bf3bf281df5c-openshift-service-ca\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.492197 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.492105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.563548 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.563500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:33.592645 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.592602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1741c8cf-c097-428a-b458-bf3bf281df5c-openshift-service-ca\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.592863 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.592661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.592863 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.592748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zjv\" (UniqueName: \"kubernetes.io/projected/1741c8cf-c097-428a-b458-bf3bf281df5c-kube-api-access-47zjv\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.592863 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:19:33.592822    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-serving-cert: secret "maas-api-e2e-mt-93ae309a-serving-cert" not found
Jul 24 23:19:33.593033 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:19:33.592899    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls podName:1741c8cf-c097-428a-b458-bf3bf281df5c nodeName:}" failed. No retries permitted until 2026-07-24 23:19:34.092882938 +0000 UTC m=+1990.726161855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls") pod "maas-api-e2e-mt-93ae309a-776fcc6666-t55lt" (UID: "1741c8cf-c097-428a-b458-bf3bf281df5c") : secret "maas-api-e2e-mt-93ae309a-serving-cert" not found
Jul 24 23:19:33.593304 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.593283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1741c8cf-c097-428a-b458-bf3bf281df5c-openshift-service-ca\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.602943 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.602912    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zjv\" (UniqueName: \"kubernetes.io/projected/1741c8cf-c097-428a-b458-bf3bf281df5c-kube-api-access-47zjv\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:33.668993 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:33.668885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:34.072646 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.072552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:34.096067 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.096030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:34.098697 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.098671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls\") pod \"maas-api-e2e-mt-93ae309a-776fcc6666-t55lt\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:34.173092 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.173057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:34.202609 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.202570    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"]
Jul 24 23:19:34.206096 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.206072    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.208985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.208961    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-93ae309a-dockercfg-wbglb\""
Jul 24 23:19:34.209124 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.209004    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-93ae309a\""
Jul 24 23:19:34.220706 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.220676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"]
Jul 24 23:19:34.263433 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.263374    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"]
Jul 24 23:19:34.267983 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.267950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.274582 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.274552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"]
Jul 24 23:19:34.297602 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.297563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ced9b95-252e-4fba-972c-443adfca1cb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk\" (UID: \"0ced9b95-252e-4fba-972c-443adfca1cb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.297811 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.297620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5556754-35f2-4ffa-84b5-e77fd7df48a3-plugins-config-volume\") pod \"payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm\" (UID: \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\") " pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.297811 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.297710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4tq\" (UniqueName: \"kubernetes.io/projected/c5556754-35f2-4ffa-84b5-e77fd7df48a3-kube-api-access-nd4tq\") pod \"payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm\" (UID: \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\") " pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.297811 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.297756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmk5w\" (UniqueName: \"kubernetes.io/projected/0ced9b95-252e-4fba-972c-443adfca1cb1-kube-api-access-rmk5w\") pod \"payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk\" (UID: \"0ced9b95-252e-4fba-972c-443adfca1cb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.390114 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.390067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:34.399419 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.399386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ced9b95-252e-4fba-972c-443adfca1cb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk\" (UID: \"0ced9b95-252e-4fba-972c-443adfca1cb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.399625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.399467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5556754-35f2-4ffa-84b5-e77fd7df48a3-plugins-config-volume\") pod \"payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm\" (UID: \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\") " pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.399625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.399558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4tq\" (UniqueName: \"kubernetes.io/projected/c5556754-35f2-4ffa-84b5-e77fd7df48a3-kube-api-access-nd4tq\") pod \"payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm\" (UID: \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\") " pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.399625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.399607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmk5w\" (UniqueName: \"kubernetes.io/projected/0ced9b95-252e-4fba-972c-443adfca1cb1-kube-api-access-rmk5w\") pod \"payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk\" (UID: \"0ced9b95-252e-4fba-972c-443adfca1cb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.400756 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.400717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5556754-35f2-4ffa-84b5-e77fd7df48a3-plugins-config-volume\") pod \"payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm\" (UID: \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\") " pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.400756 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.400713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ced9b95-252e-4fba-972c-443adfca1cb1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk\" (UID: \"0ced9b95-252e-4fba-972c-443adfca1cb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.412494 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.412456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4tq\" (UniqueName: \"kubernetes.io/projected/c5556754-35f2-4ffa-84b5-e77fd7df48a3-kube-api-access-nd4tq\") pod \"payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm\" (UID: \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\") " pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.412793 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.412773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmk5w\" (UniqueName: \"kubernetes.io/projected/0ced9b95-252e-4fba-972c-443adfca1cb1-kube-api-access-rmk5w\") pod \"payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk\" (UID: \"0ced9b95-252e-4fba-972c-443adfca1cb1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.517576 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.517513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.567649 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.567613    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"]
Jul 24 23:19:34.579749 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.579714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.673692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.673664    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"]
Jul 24 23:19:34.676786 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:19:34.676756    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5556754_35f2_4ffa_84b5_e77fd7df48a3.slice/crio-606f2f26674b410153b7a50545d5f56372f59e925d0adee5b5b8bb14e774426b WatchSource:0}: Error finding container 606f2f26674b410153b7a50545d5f56372f59e925d0adee5b5b8bb14e774426b: Status 404 returned error can't find the container with id 606f2f26674b410153b7a50545d5f56372f59e925d0adee5b5b8bb14e774426b
Jul 24 23:19:34.743596 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.743558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"]
Jul 24 23:19:34.745639 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:19:34.745607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ced9b95_252e_4fba_972c_443adfca1cb1.slice/crio-29053748f3ec68d96caf9e4fb4a85bfe67a6525aa70df3c35a6f39aa2d4a03dc WatchSource:0}: Error finding container 29053748f3ec68d96caf9e4fb4a85bfe67a6525aa70df3c35a6f39aa2d4a03dc: Status 404 returned error can't find the container with id 29053748f3ec68d96caf9e4fb4a85bfe67a6525aa70df3c35a6f39aa2d4a03dc
Jul 24 23:19:34.874066 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.874032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt" event={"ID":"1741c8cf-c097-428a-b458-bf3bf281df5c","Type":"ContainerStarted","Data":"1c52479e9dcb720a7d4ed6490752bb5675f727d813095867a9d124d23754c622"}
Jul 24 23:19:34.875483 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.875450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk" event={"ID":"0ced9b95-252e-4fba-972c-443adfca1cb1","Type":"ContainerStarted","Data":"da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98"}
Jul 24 23:19:34.875659 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.875508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk" event={"ID":"0ced9b95-252e-4fba-972c-443adfca1cb1","Type":"ContainerStarted","Data":"29053748f3ec68d96caf9e4fb4a85bfe67a6525aa70df3c35a6f39aa2d4a03dc"}
Jul 24 23:19:34.875659 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.875551    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:34.877078 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.877047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm" event={"ID":"c5556754-35f2-4ffa-84b5-e77fd7df48a3","Type":"ContainerStarted","Data":"a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5"}
Jul 24 23:19:34.877262 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.877096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm" event={"ID":"c5556754-35f2-4ffa-84b5-e77fd7df48a3","Type":"ContainerStarted","Data":"606f2f26674b410153b7a50545d5f56372f59e925d0adee5b5b8bb14e774426b"}
Jul 24 23:19:34.877262 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.877125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:34.893974 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.893871    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk" podStartSLOduration=0.893849087 podStartE2EDuration="893.849087ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:19:34.892792389 +0000 UTC m=+1991.526071330" watchObservedRunningTime="2026-07-24 23:19:34.893849087 +0000 UTC m=+1991.527128004"
Jul 24 23:19:34.913125 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:34.913065    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm" podStartSLOduration=0.913048522 podStartE2EDuration="913.048522ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:19:34.911395989 +0000 UTC m=+1991.544674933" watchObservedRunningTime="2026-07-24 23:19:34.913048522 +0000 UTC m=+1991.546327462"
Jul 24 23:19:35.884143 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:35.884104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt" event={"ID":"1741c8cf-c097-428a-b458-bf3bf281df5c","Type":"ContainerStarted","Data":"90ceb9bc020db1d99679e83b6ebd886a862a8e6f55c5365ce57a258231d4b981"}
Jul 24 23:19:35.884593 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:35.884429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:35.905539 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:35.905460    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt" podStartSLOduration=2.271556085 podStartE2EDuration="2.905440781s" totalImagesPullingTime="633.884696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:19:35.904217686 +0000 UTC m=+1992.537496640" watchObservedRunningTime="2026-07-24 23:19:35.905440781 +0000 UTC m=+1992.538719699"
Jul 24 23:19:36.844812 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:36.844776    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:19:37.888787 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:37.888753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:38.538234 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:38.538200    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:38.638315 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:38.638283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:38.971410 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:38.971374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:39.779268 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:39.779236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:41.895658 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:41.895629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:44.918493 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:44.918449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:45.020817 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.020787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:45.346559 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.346509    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"]
Jul 24 23:19:45.351270 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.351249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"
Jul 24 23:19:45.354917 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.354898    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:19:45.359561 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.358980    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"]
Jul 24 23:19:45.385769 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.385731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82acc593-499a-470a-91d3-89d1dcb93c6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw\" (UID: \"82acc593-499a-470a-91d3-89d1dcb93c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"
Jul 24 23:19:45.486953 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.486909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82acc593-499a-470a-91d3-89d1dcb93c6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw\" (UID: \"82acc593-499a-470a-91d3-89d1dcb93c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"
Jul 24 23:19:45.488114 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.488074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82acc593-499a-470a-91d3-89d1dcb93c6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw\" (UID: \"82acc593-499a-470a-91d3-89d1dcb93c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"
Jul 24 23:19:45.663777 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.663682    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"
Jul 24 23:19:45.807215 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.807177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"]
Jul 24 23:19:45.809720 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:19:45.809687    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82acc593_499a_470a_91d3_89d1dcb93c6c.slice/crio-ff6ac9e348a40694a33d9536e525e47203150b0b0e1a2e31df0468a272d8c8c1 WatchSource:0}: Error finding container ff6ac9e348a40694a33d9536e525e47203150b0b0e1a2e31df0468a272d8c8c1: Status 404 returned error can't find the container with id ff6ac9e348a40694a33d9536e525e47203150b0b0e1a2e31df0468a272d8c8c1
Jul 24 23:19:45.886342 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.886310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:45.886569 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.886382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:45.937548 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:45.937437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw" event={"ID":"82acc593-499a-470a-91d3-89d1dcb93c6c","Type":"ContainerStarted","Data":"ff6ac9e348a40694a33d9536e525e47203150b0b0e1a2e31df0468a272d8c8c1"}
Jul 24 23:19:46.944827 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:46.944785    2563 generic.go:356] "Generic (PLEG): container finished" podID="82acc593-499a-470a-91d3-89d1dcb93c6c" containerID="54777d87e9f877f798b348c1a58f08cfb27343446a8aa194cc665691d45b0649" exitCode=0
Jul 24 23:19:46.945239 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:46.944833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw" event={"ID":"82acc593-499a-470a-91d3-89d1dcb93c6c","Type":"ContainerDied","Data":"54777d87e9f877f798b348c1a58f08cfb27343446a8aa194cc665691d45b0649"}
Jul 24 23:19:48.085996 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:48.085971    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"
Jul 24 23:19:48.110295 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:48.110263    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82acc593-499a-470a-91d3-89d1dcb93c6c-maas-api-service-ca\") pod \"82acc593-499a-470a-91d3-89d1dcb93c6c\" (UID: \"82acc593-499a-470a-91d3-89d1dcb93c6c\") "
Jul 24 23:19:48.110679 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:48.110651    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82acc593-499a-470a-91d3-89d1dcb93c6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82acc593-499a-470a-91d3-89d1dcb93c6c" (UID: "82acc593-499a-470a-91d3-89d1dcb93c6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:19:48.211405 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:48.211316    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82acc593-499a-470a-91d3-89d1dcb93c6c-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:19:48.955026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:48.954995    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"
Jul 24 23:19:48.955026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:48.955008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw" event={"ID":"82acc593-499a-470a-91d3-89d1dcb93c6c","Type":"ContainerDied","Data":"ff6ac9e348a40694a33d9536e525e47203150b0b0e1a2e31df0468a272d8c8c1"}
Jul 24 23:19:48.955244 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:48.955052    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff6ac9e348a40694a33d9536e525e47203150b0b0e1a2e31df0468a272d8c8c1"
Jul 24 23:19:55.866394 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.866358    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"]
Jul 24 23:19:55.866925 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.866619    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt" podUID="1741c8cf-c097-428a-b458-bf3bf281df5c" containerName="maas-api" containerID="cri-o://90ceb9bc020db1d99679e83b6ebd886a862a8e6f55c5365ce57a258231d4b981" gracePeriod=30
Jul 24 23:19:55.899239 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.899204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:55.930551 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.930498    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"]
Jul 24 23:19:55.930792 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.930760    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm" podUID="c5556754-35f2-4ffa-84b5-e77fd7df48a3" containerName="payload-processing" containerID="cri-o://a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5" gracePeriod=30
Jul 24 23:19:55.946187 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.946146    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"]
Jul 24 23:19:55.946492 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.946456    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk" podUID="0ced9b95-252e-4fba-972c-443adfca1cb1" containerName="payload-pre-processing" containerID="cri-o://da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98" gracePeriod=30
Jul 24 23:19:55.991107 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.991070    2563 generic.go:356] "Generic (PLEG): container finished" podID="1741c8cf-c097-428a-b458-bf3bf281df5c" containerID="90ceb9bc020db1d99679e83b6ebd886a862a8e6f55c5365ce57a258231d4b981" exitCode=0
Jul 24 23:19:55.991278 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.991141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt" event={"ID":"1741c8cf-c097-428a-b458-bf3bf281df5c","Type":"ContainerDied","Data":"90ceb9bc020db1d99679e83b6ebd886a862a8e6f55c5365ce57a258231d4b981"}
Jul 24 23:19:55.996625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:55.996601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:56.122401 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.122300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:19:56.204954 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.204925    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:56.238720 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.238691    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:56.249886 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.249858    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:56.281592 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.281559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls\") pod \"1741c8cf-c097-428a-b458-bf3bf281df5c\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") "
Jul 24 23:19:56.281751 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.281617    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47zjv\" (UniqueName: \"kubernetes.io/projected/1741c8cf-c097-428a-b458-bf3bf281df5c-kube-api-access-47zjv\") pod \"1741c8cf-c097-428a-b458-bf3bf281df5c\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") "
Jul 24 23:19:56.281751 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.281684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd4tq\" (UniqueName: \"kubernetes.io/projected/c5556754-35f2-4ffa-84b5-e77fd7df48a3-kube-api-access-nd4tq\") pod \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\" (UID: \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\") "
Jul 24 23:19:56.281751 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.281729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1741c8cf-c097-428a-b458-bf3bf281df5c-openshift-service-ca\") pod \"1741c8cf-c097-428a-b458-bf3bf281df5c\" (UID: \"1741c8cf-c097-428a-b458-bf3bf281df5c\") "
Jul 24 23:19:56.281945 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.281753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5556754-35f2-4ffa-84b5-e77fd7df48a3-plugins-config-volume\") pod \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\" (UID: \"c5556754-35f2-4ffa-84b5-e77fd7df48a3\") "
Jul 24 23:19:56.281945 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.281789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ced9b95-252e-4fba-972c-443adfca1cb1-plugins-config-volume\") pod \"0ced9b95-252e-4fba-972c-443adfca1cb1\" (UID: \"0ced9b95-252e-4fba-972c-443adfca1cb1\") "
Jul 24 23:19:56.281945 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.281850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmk5w\" (UniqueName: \"kubernetes.io/projected/0ced9b95-252e-4fba-972c-443adfca1cb1-kube-api-access-rmk5w\") pod \"0ced9b95-252e-4fba-972c-443adfca1cb1\" (UID: \"0ced9b95-252e-4fba-972c-443adfca1cb1\") "
Jul 24 23:19:56.282320 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.282248    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5556754-35f2-4ffa-84b5-e77fd7df48a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5556754-35f2-4ffa-84b5-e77fd7df48a3" (UID: "c5556754-35f2-4ffa-84b5-e77fd7df48a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:19:56.282420 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.282323    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ced9b95-252e-4fba-972c-443adfca1cb1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ced9b95-252e-4fba-972c-443adfca1cb1" (UID: "0ced9b95-252e-4fba-972c-443adfca1cb1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:19:56.282502 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.282478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1741c8cf-c097-428a-b458-bf3bf281df5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1741c8cf-c097-428a-b458-bf3bf281df5c" (UID: "1741c8cf-c097-428a-b458-bf3bf281df5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:19:56.283893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.283864    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1741c8cf-c097-428a-b458-bf3bf281df5c" (UID: "1741c8cf-c097-428a-b458-bf3bf281df5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:19:56.283996 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.283908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1741c8cf-c097-428a-b458-bf3bf281df5c-kube-api-access-47zjv" (OuterVolumeSpecName: "kube-api-access-47zjv") pod "1741c8cf-c097-428a-b458-bf3bf281df5c" (UID: "1741c8cf-c097-428a-b458-bf3bf281df5c"). InnerVolumeSpecName "kube-api-access-47zjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:19:56.283996 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.283916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5556754-35f2-4ffa-84b5-e77fd7df48a3-kube-api-access-nd4tq" (OuterVolumeSpecName: "kube-api-access-nd4tq") pod "c5556754-35f2-4ffa-84b5-e77fd7df48a3" (UID: "c5556754-35f2-4ffa-84b5-e77fd7df48a3"). InnerVolumeSpecName "kube-api-access-nd4tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:19:56.284342 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.284325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ced9b95-252e-4fba-972c-443adfca1cb1-kube-api-access-rmk5w" (OuterVolumeSpecName: "kube-api-access-rmk5w") pod "0ced9b95-252e-4fba-972c-443adfca1cb1" (UID: "0ced9b95-252e-4fba-972c-443adfca1cb1"). InnerVolumeSpecName "kube-api-access-rmk5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:19:56.383026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.382933    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1741c8cf-c097-428a-b458-bf3bf281df5c-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:19:56.383026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.382967    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5556754-35f2-4ffa-84b5-e77fd7df48a3-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:19:56.383026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.382978    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ced9b95-252e-4fba-972c-443adfca1cb1-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:19:56.383026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.382987    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmk5w\" (UniqueName: \"kubernetes.io/projected/0ced9b95-252e-4fba-972c-443adfca1cb1-kube-api-access-rmk5w\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:19:56.383026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.382998    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1741c8cf-c097-428a-b458-bf3bf281df5c-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:19:56.383026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.383007    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zjv\" (UniqueName: \"kubernetes.io/projected/1741c8cf-c097-428a-b458-bf3bf281df5c-kube-api-access-47zjv\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:19:56.383026 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.383016    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd4tq\" (UniqueName: \"kubernetes.io/projected/c5556754-35f2-4ffa-84b5-e77fd7df48a3-kube-api-access-nd4tq\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:19:56.997001 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.996966    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"
Jul 24 23:19:56.997001 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.996983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt" event={"ID":"1741c8cf-c097-428a-b458-bf3bf281df5c","Type":"ContainerDied","Data":"1c52479e9dcb720a7d4ed6490752bb5675f727d813095867a9d124d23754c622"}
Jul 24 23:19:56.997581 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.997048    2563 scope.go:117] "RemoveContainer" containerID="90ceb9bc020db1d99679e83b6ebd886a862a8e6f55c5365ce57a258231d4b981"
Jul 24 23:19:56.998442 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.998418    2563 generic.go:356] "Generic (PLEG): container finished" podID="0ced9b95-252e-4fba-972c-443adfca1cb1" containerID="da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98" exitCode=0
Jul 24 23:19:56.998572 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.998470    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"
Jul 24 23:19:56.998572 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.998503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk" event={"ID":"0ced9b95-252e-4fba-972c-443adfca1cb1","Type":"ContainerDied","Data":"da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98"}
Jul 24 23:19:56.998687 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.998574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk" event={"ID":"0ced9b95-252e-4fba-972c-443adfca1cb1","Type":"ContainerDied","Data":"29053748f3ec68d96caf9e4fb4a85bfe67a6525aa70df3c35a6f39aa2d4a03dc"}
Jul 24 23:19:56.999971 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:56.999946    2563 generic.go:356] "Generic (PLEG): container finished" podID="c5556754-35f2-4ffa-84b5-e77fd7df48a3" containerID="a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5" exitCode=0
Jul 24 23:19:57.000084 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.000009    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"
Jul 24 23:19:57.000084 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.000009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm" event={"ID":"c5556754-35f2-4ffa-84b5-e77fd7df48a3","Type":"ContainerDied","Data":"a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5"}
Jul 24 23:19:57.000209 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.000083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm" event={"ID":"c5556754-35f2-4ffa-84b5-e77fd7df48a3","Type":"ContainerDied","Data":"606f2f26674b410153b7a50545d5f56372f59e925d0adee5b5b8bb14e774426b"}
Jul 24 23:19:57.007669 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.007602    2563 scope.go:117] "RemoveContainer" containerID="da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98"
Jul 24 23:19:57.017610 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.017580    2563 scope.go:117] "RemoveContainer" containerID="da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98"
Jul 24 23:19:57.017942 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:19:57.017921    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98\": container with ID starting with da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98 not found: ID does not exist" containerID="da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98"
Jul 24 23:19:57.018030 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.017955    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98"} err="failed to get container status \"da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98\": rpc error: code = NotFound desc = could not find container \"da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98\": container with ID starting with da6a03728e65c31bbd7fc5a42982bc772ef2da53e1737413b8a43447b56bea98 not found: ID does not exist"
Jul 24 23:19:57.018030 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.017981    2563 scope.go:117] "RemoveContainer" containerID="a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5"
Jul 24 23:19:57.024848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.024817    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"]
Jul 24 23:19:57.026392 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.026378    2563 scope.go:117] "RemoveContainer" containerID="a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5"
Jul 24 23:19:57.026693 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:19:57.026674    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5\": container with ID starting with a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5 not found: ID does not exist" containerID="a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5"
Jul 24 23:19:57.026770 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.026702    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5"} err="failed to get container status \"a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5\": rpc error: code = NotFound desc = could not find container \"a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5\": container with ID starting with a613728cead01c903c2fa962ddb819da70ad5ba927c589b00259084cf74421d5 not found: ID does not exist"
Jul 24 23:19:57.029325 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.029298    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-93ae309a-6cdd85fdf6-676jk"]
Jul 24 23:19:57.040646 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.040616    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"]
Jul 24 23:19:57.045105 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.045061    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-93ae309a-776fcc6666-t55lt"]
Jul 24 23:19:57.060150 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.060109    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"]
Jul 24 23:19:57.063942 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.063910    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-93ae309a-7894fdc76b-qtswm"]
Jul 24 23:19:57.842167 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.842132    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ced9b95-252e-4fba-972c-443adfca1cb1" path="/var/lib/kubelet/pods/0ced9b95-252e-4fba-972c-443adfca1cb1/volumes"
Jul 24 23:19:57.842545 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.842514    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1741c8cf-c097-428a-b458-bf3bf281df5c" path="/var/lib/kubelet/pods/1741c8cf-c097-428a-b458-bf3bf281df5c/volumes"
Jul 24 23:19:57.842869 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:57.842857    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5556754-35f2-4ffa-84b5-e77fd7df48a3" path="/var/lib/kubelet/pods/c5556754-35f2-4ffa-84b5-e77fd7df48a3/volumes"
Jul 24 23:19:58.739599 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:19:58.739556    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:20:00.514951 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:00.514913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:00.515473 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:00.515346    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"]
Jul 24 23:20:00.517550 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:00.517512    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-93ae309a-bz8kw"]
Jul 24 23:20:00.902203 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:00.902170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:01.840849 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:01.840813    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82acc593-499a-470a-91d3-89d1dcb93c6c" path="/var/lib/kubelet/pods/82acc593-499a-470a-91d3-89d1dcb93c6c/volumes"
Jul 24 23:20:06.622266 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:06.622230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:06.793697 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:06.793662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:06.932666 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:06.932578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:07.160567 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:07.160506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:07.271130 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:07.271036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:07.367890 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:07.367860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:07.480668 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:07.480636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:13.228230 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:13.228198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:13.334643 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:13.334587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:13.428663 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:13.428623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:13.579424 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:13.579340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:19.536247 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:19.536212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:19.636768 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:19.636730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:26.317440 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:26.317403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:26.643482 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:26.643448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:26.912053 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:26.911969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:29.694196 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.694155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:29.783753 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.783482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:29.784941 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.784911    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"]
Jul 24 23:20:29.785496 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5556754-35f2-4ffa-84b5-e77fd7df48a3" containerName="payload-processing"
Jul 24 23:20:29.785577 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785500    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5556754-35f2-4ffa-84b5-e77fd7df48a3" containerName="payload-processing"
Jul 24 23:20:29.785577 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785532    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ced9b95-252e-4fba-972c-443adfca1cb1" containerName="payload-pre-processing"
Jul 24 23:20:29.785577 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785542    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ced9b95-252e-4fba-972c-443adfca1cb1" containerName="payload-pre-processing"
Jul 24 23:20:29.785577 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785554    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1741c8cf-c097-428a-b458-bf3bf281df5c" containerName="maas-api"
Jul 24 23:20:29.785577 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785563    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1741c8cf-c097-428a-b458-bf3bf281df5c" containerName="maas-api"
Jul 24 23:20:29.785848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785580    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82acc593-499a-470a-91d3-89d1dcb93c6c" containerName="revoke-keys"
Jul 24 23:20:29.785848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785590    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="82acc593-499a-470a-91d3-89d1dcb93c6c" containerName="revoke-keys"
Jul 24 23:20:29.785848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785688    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1741c8cf-c097-428a-b458-bf3bf281df5c" containerName="maas-api"
Jul 24 23:20:29.785848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785700    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5556754-35f2-4ffa-84b5-e77fd7df48a3" containerName="payload-processing"
Jul 24 23:20:29.785848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785711    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ced9b95-252e-4fba-972c-443adfca1cb1" containerName="payload-pre-processing"
Jul 24 23:20:29.785848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.785721    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="82acc593-499a-470a-91d3-89d1dcb93c6c" containerName="revoke-keys"
Jul 24 23:20:29.789017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.788986    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:29.791979 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.791938    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:20:29.792129 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.791980    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:20:29.793176 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.793157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4813d775-serving-cert\""
Jul 24 23:20:29.793233 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.793172    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:20:29.800202 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.800171    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"]
Jul 24 23:20:29.863697 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.863664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:29.962981 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.962897    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:29.983033 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.983001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe657b3b-fddd-414b-aff3-a8d971dc9a60-maas-api-tls\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:29.983209 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.983096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzr5g\" (UniqueName: \"kubernetes.io/projected/fe657b3b-fddd-414b-aff3-a8d971dc9a60-kube-api-access-dzr5g\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:29.983209 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:29.983128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe657b3b-fddd-414b-aff3-a8d971dc9a60-openshift-service-ca\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:30.084429 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.084388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe657b3b-fddd-414b-aff3-a8d971dc9a60-openshift-service-ca\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:30.084638 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.084459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe657b3b-fddd-414b-aff3-a8d971dc9a60-maas-api-tls\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:30.084638 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.084568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzr5g\" (UniqueName: \"kubernetes.io/projected/fe657b3b-fddd-414b-aff3-a8d971dc9a60-kube-api-access-dzr5g\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:30.085187 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.085164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe657b3b-fddd-414b-aff3-a8d971dc9a60-openshift-service-ca\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:30.086915 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.086892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe657b3b-fddd-414b-aff3-a8d971dc9a60-maas-api-tls\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:30.097810 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.097772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzr5g\" (UniqueName: \"kubernetes.io/projected/fe657b3b-fddd-414b-aff3-a8d971dc9a60-kube-api-access-dzr5g\") pod \"maas-api-e2e-mt-4813d775-c44ff6544-t55sc\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:30.102602 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.102565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:30.246099 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.246008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"]
Jul 24 23:20:30.249540 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:20:30.249492    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe657b3b_fddd_414b_aff3_a8d971dc9a60.slice/crio-8e8f632913fe76f4d9beb90c0b1a6247e41d1ad528cb90b4c8dcc00e382fae16 WatchSource:0}: Error finding container 8e8f632913fe76f4d9beb90c0b1a6247e41d1ad528cb90b4c8dcc00e382fae16: Status 404 returned error can't find the container with id 8e8f632913fe76f4d9beb90c0b1a6247e41d1ad528cb90b4c8dcc00e382fae16
Jul 24 23:20:30.251286 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.251269    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 23:20:30.384801 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.384774    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:30.488121 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.488090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:30.508754 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.508666    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"]
Jul 24 23:20:30.513306 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.513287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:30.516063 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.516036    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4813d775\""
Jul 24 23:20:30.516424 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.516396    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4813d775-dockercfg-8hg4t\""
Jul 24 23:20:30.525327 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.525298    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"]
Jul 24 23:20:30.526803 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.526781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"]
Jul 24 23:20:30.529878 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.529858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:30.541594 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.541569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"]
Jul 24 23:20:30.689115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.689079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8498bb9-359d-4c21-b59b-f5eed6d923dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4\" (UID: \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:30.689115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.689117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k9dr\" (UniqueName: \"kubernetes.io/projected/f8498bb9-359d-4c21-b59b-f5eed6d923dd-kube-api-access-8k9dr\") pod \"payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4\" (UID: \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:30.689349 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.689159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159cae97-05f0-4465-ad1e-86ca0c78f9ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf\" (UID: \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\") " pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:30.689349 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.689204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzsk\" (UniqueName: \"kubernetes.io/projected/159cae97-05f0-4465-ad1e-86ca0c78f9ec-kube-api-access-6qzsk\") pod \"payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf\" (UID: \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\") " pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:30.790484 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.790376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8498bb9-359d-4c21-b59b-f5eed6d923dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4\" (UID: \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:30.790484 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.790419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k9dr\" (UniqueName: \"kubernetes.io/projected/f8498bb9-359d-4c21-b59b-f5eed6d923dd-kube-api-access-8k9dr\") pod \"payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4\" (UID: \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:30.791052 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.790578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159cae97-05f0-4465-ad1e-86ca0c78f9ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf\" (UID: \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\") " pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:30.791052 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.790656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzsk\" (UniqueName: \"kubernetes.io/projected/159cae97-05f0-4465-ad1e-86ca0c78f9ec-kube-api-access-6qzsk\") pod \"payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf\" (UID: \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\") " pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:30.791359 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.791336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159cae97-05f0-4465-ad1e-86ca0c78f9ec-plugins-config-volume\") pod \"payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf\" (UID: \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\") " pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:30.791435 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.791336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8498bb9-359d-4c21-b59b-f5eed6d923dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4\" (UID: \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:30.799736 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.799699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzsk\" (UniqueName: \"kubernetes.io/projected/159cae97-05f0-4465-ad1e-86ca0c78f9ec-kube-api-access-6qzsk\") pod \"payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf\" (UID: \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\") " pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:30.800439 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.800413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k9dr\" (UniqueName: \"kubernetes.io/projected/f8498bb9-359d-4c21-b59b-f5eed6d923dd-kube-api-access-8k9dr\") pod \"payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4\" (UID: \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:30.825327 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.825287    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:30.841713 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.841688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:30.981755 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:30.981725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"]
Jul 24 23:20:30.984611 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:20:30.984507    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8498bb9_359d_4c21_b59b_f5eed6d923dd.slice/crio-6f8967448766b469780018a1615337f990cb55a1f4deee52b715f94f63bd8202 WatchSource:0}: Error finding container 6f8967448766b469780018a1615337f990cb55a1f4deee52b715f94f63bd8202: Status 404 returned error can't find the container with id 6f8967448766b469780018a1615337f990cb55a1f4deee52b715f94f63bd8202
Jul 24 23:20:31.003183 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.003151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"]
Jul 24 23:20:31.005860 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:20:31.005826    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159cae97_05f0_4465_ad1e_86ca0c78f9ec.slice/crio-bd4513c310253398703a94605cd2df8c3ca76e2b1bfee0e465b77e88eeaabe71 WatchSource:0}: Error finding container bd4513c310253398703a94605cd2df8c3ca76e2b1bfee0e465b77e88eeaabe71: Status 404 returned error can't find the container with id bd4513c310253398703a94605cd2df8c3ca76e2b1bfee0e465b77e88eeaabe71
Jul 24 23:20:31.180710 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.180674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4" event={"ID":"f8498bb9-359d-4c21-b59b-f5eed6d923dd","Type":"ContainerStarted","Data":"782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768"}
Jul 24 23:20:31.180885 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.180726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4" event={"ID":"f8498bb9-359d-4c21-b59b-f5eed6d923dd","Type":"ContainerStarted","Data":"6f8967448766b469780018a1615337f990cb55a1f4deee52b715f94f63bd8202"}
Jul 24 23:20:31.180885 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.180765    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:31.182200 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.182175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf" event={"ID":"159cae97-05f0-4465-ad1e-86ca0c78f9ec","Type":"ContainerStarted","Data":"86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9"}
Jul 24 23:20:31.182342 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.182221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf" event={"ID":"159cae97-05f0-4465-ad1e-86ca0c78f9ec","Type":"ContainerStarted","Data":"bd4513c310253398703a94605cd2df8c3ca76e2b1bfee0e465b77e88eeaabe71"}
Jul 24 23:20:31.182342 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.182246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:31.183678 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.183659    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc" event={"ID":"fe657b3b-fddd-414b-aff3-a8d971dc9a60","Type":"ContainerStarted","Data":"9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc"}
Jul 24 23:20:31.183777 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.183693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc" event={"ID":"fe657b3b-fddd-414b-aff3-a8d971dc9a60","Type":"ContainerStarted","Data":"8e8f632913fe76f4d9beb90c0b1a6247e41d1ad528cb90b4c8dcc00e382fae16"}
Jul 24 23:20:31.183817 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.183777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:31.201333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.201278    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4" podStartSLOduration=1.201261481 podStartE2EDuration="1.201261481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:20:31.198634144 +0000 UTC m=+2047.831913085" watchObservedRunningTime="2026-07-24 23:20:31.201261481 +0000 UTC m=+2047.834540420"
Jul 24 23:20:31.217123 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.217036    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc" podStartSLOduration=1.860315916 podStartE2EDuration="2.217019516s" totalImagesPullingTime="356.7036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:20:31.216084678 +0000 UTC m=+2047.849363624" watchObservedRunningTime="2026-07-24 23:20:31.217019516 +0000 UTC m=+2047.850298508"
Jul 24 23:20:31.234412 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:31.234345    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf" podStartSLOduration=1.234322028 podStartE2EDuration="1.234322028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:20:31.233834851 +0000 UTC m=+2047.867113818" watchObservedRunningTime="2026-07-24 23:20:31.234322028 +0000 UTC m=+2047.867600968"
Jul 24 23:20:34.193316 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:34.193265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:34.432824 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:34.432788    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:20:37.196160 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:37.196130    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:20:42.192087 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:42.192049    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:20:42.192476 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:42.192142    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:20:52.009961 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:52.009879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:20:55.039504 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:20:55.039462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:21:08.942602 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:08.942560    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:21:10.621352 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:10.621314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:21:13.644369 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:13.644331    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"]
Jul 24 23:21:13.648847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:13.648826    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"
Jul 24 23:21:13.652931 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:13.652911    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:21:13.657933 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:13.657905    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"]
Jul 24 23:21:13.757500 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:13.757463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8fb8a6-dfe8-4a91-851a-5f6bce1245db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4813d775-8vtcp\" (UID: \"9a8fb8a6-dfe8-4a91-851a-5f6bce1245db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"
Jul 24 23:21:13.858593 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:13.858552    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8fb8a6-dfe8-4a91-851a-5f6bce1245db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4813d775-8vtcp\" (UID: \"9a8fb8a6-dfe8-4a91-851a-5f6bce1245db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"
Jul 24 23:21:13.859194 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:13.859172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8fb8a6-dfe8-4a91-851a-5f6bce1245db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4813d775-8vtcp\" (UID: \"9a8fb8a6-dfe8-4a91-851a-5f6bce1245db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"
Jul 24 23:21:13.961560 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:13.961439    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"
Jul 24 23:21:14.102071 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:14.102046    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"]
Jul 24 23:21:14.104554 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:21:14.104508    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8fb8a6_dfe8_4a91_851a_5f6bce1245db.slice/crio-c226480817ee53850477fbdd5fae766be99838918e97840fe8bbcec6772b99a8 WatchSource:0}: Error finding container c226480817ee53850477fbdd5fae766be99838918e97840fe8bbcec6772b99a8: Status 404 returned error can't find the container with id c226480817ee53850477fbdd5fae766be99838918e97840fe8bbcec6772b99a8
Jul 24 23:21:14.414979 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:14.414945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp" event={"ID":"9a8fb8a6-dfe8-4a91-851a-5f6bce1245db","Type":"ContainerStarted","Data":"c226480817ee53850477fbdd5fae766be99838918e97840fe8bbcec6772b99a8"}
Jul 24 23:21:15.421959 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:15.421926    2563 generic.go:356] "Generic (PLEG): container finished" podID="9a8fb8a6-dfe8-4a91-851a-5f6bce1245db" containerID="5bf586a27bbb7174f79666b5a5c00d4fcf2d8afb429651b5a4a043bb275cef46" exitCode=0
Jul 24 23:21:15.422352 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:15.421991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp" event={"ID":"9a8fb8a6-dfe8-4a91-851a-5f6bce1245db","Type":"ContainerDied","Data":"5bf586a27bbb7174f79666b5a5c00d4fcf2d8afb429651b5a4a043bb275cef46"}
Jul 24 23:21:16.565858 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:16.565817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"
Jul 24 23:21:16.683139 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:16.683041    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8fb8a6-dfe8-4a91-851a-5f6bce1245db-maas-api-service-ca\") pod \"9a8fb8a6-dfe8-4a91-851a-5f6bce1245db\" (UID: \"9a8fb8a6-dfe8-4a91-851a-5f6bce1245db\") "
Jul 24 23:21:16.683424 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:16.683399    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8fb8a6-dfe8-4a91-851a-5f6bce1245db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a8fb8a6-dfe8-4a91-851a-5f6bce1245db" (UID: "9a8fb8a6-dfe8-4a91-851a-5f6bce1245db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:21:16.784219 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:16.784183    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8fb8a6-dfe8-4a91-851a-5f6bce1245db-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:21:17.433145 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:17.433113    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"
Jul 24 23:21:17.433360 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:17.433156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp" event={"ID":"9a8fb8a6-dfe8-4a91-851a-5f6bce1245db","Type":"ContainerDied","Data":"c226480817ee53850477fbdd5fae766be99838918e97840fe8bbcec6772b99a8"}
Jul 24 23:21:17.433360 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:17.433202    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c226480817ee53850477fbdd5fae766be99838918e97840fe8bbcec6772b99a8"
Jul 24 23:21:23.888167 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:23.888138    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:21:23.890828 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:23.890802    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:21:24.003599 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:24.003553    2563 scope.go:117] "RemoveContainer" containerID="3f342a74ba1a5e84380a76d80c8abedb8f7e13cc054b01879bbb572f41943c8e"
Jul 24 23:21:27.628629 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:27.628575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:21:29.122513 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.122483    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"]
Jul 24 23:21:29.122954 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.122718    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc" podUID="fe657b3b-fddd-414b-aff3-a8d971dc9a60" containerName="maas-api" containerID="cri-o://9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc" gracePeriod=30
Jul 24 23:21:29.152408 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.152378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:21:29.175534 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.175493    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"]
Jul 24 23:21:29.175837 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.175774    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf" podUID="159cae97-05f0-4465-ad1e-86ca0c78f9ec" containerName="payload-processing" containerID="cri-o://86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9" gracePeriod=30
Jul 24 23:21:29.182776 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.182640    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"]
Jul 24 23:21:29.182930 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.182875    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4" podUID="f8498bb9-359d-4c21-b59b-f5eed6d923dd" containerName="payload-pre-processing" containerID="cri-o://782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768" gracePeriod=30
Jul 24 23:21:29.214107 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.214076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:21:29.270367 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.270333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:21:29.443339 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.443311    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:21:29.484103 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.484080    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:21:29.487330 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.487306    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:21:29.496984 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.496946    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159cae97-05f0-4465-ad1e-86ca0c78f9ec-plugins-config-volume\") pod \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\" (UID: \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\") "
Jul 24 23:21:29.497173 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.497105    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qzsk\" (UniqueName: \"kubernetes.io/projected/159cae97-05f0-4465-ad1e-86ca0c78f9ec-kube-api-access-6qzsk\") pod \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\" (UID: \"159cae97-05f0-4465-ad1e-86ca0c78f9ec\") "
Jul 24 23:21:29.497245 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.497191    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe657b3b-fddd-414b-aff3-a8d971dc9a60-openshift-service-ca\") pod \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") "
Jul 24 23:21:29.497245 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.497221    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k9dr\" (UniqueName: \"kubernetes.io/projected/f8498bb9-359d-4c21-b59b-f5eed6d923dd-kube-api-access-8k9dr\") pod \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\" (UID: \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\") "
Jul 24 23:21:29.497355 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.497249    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8498bb9-359d-4c21-b59b-f5eed6d923dd-plugins-config-volume\") pod \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\" (UID: \"f8498bb9-359d-4c21-b59b-f5eed6d923dd\") "
Jul 24 23:21:29.497355 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.497290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe657b3b-fddd-414b-aff3-a8d971dc9a60-maas-api-tls\") pod \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") "
Jul 24 23:21:29.497355 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.497328    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzr5g\" (UniqueName: \"kubernetes.io/projected/fe657b3b-fddd-414b-aff3-a8d971dc9a60-kube-api-access-dzr5g\") pod \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\" (UID: \"fe657b3b-fddd-414b-aff3-a8d971dc9a60\") "
Jul 24 23:21:29.500389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.497607    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe657b3b-fddd-414b-aff3-a8d971dc9a60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe657b3b-fddd-414b-aff3-a8d971dc9a60" (UID: "fe657b3b-fddd-414b-aff3-a8d971dc9a60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:21:29.500389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.497693    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8498bb9-359d-4c21-b59b-f5eed6d923dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8498bb9-359d-4c21-b59b-f5eed6d923dd" (UID: "f8498bb9-359d-4c21-b59b-f5eed6d923dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:21:29.500389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.498577    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159cae97-05f0-4465-ad1e-86ca0c78f9ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "159cae97-05f0-4465-ad1e-86ca0c78f9ec" (UID: "159cae97-05f0-4465-ad1e-86ca0c78f9ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:21:29.500389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.500241    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159cae97-05f0-4465-ad1e-86ca0c78f9ec-kube-api-access-6qzsk" (OuterVolumeSpecName: "kube-api-access-6qzsk") pod "159cae97-05f0-4465-ad1e-86ca0c78f9ec" (UID: "159cae97-05f0-4465-ad1e-86ca0c78f9ec"). InnerVolumeSpecName "kube-api-access-6qzsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:21:29.503023 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.502718    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe657b3b-fddd-414b-aff3-a8d971dc9a60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe657b3b-fddd-414b-aff3-a8d971dc9a60" (UID: "fe657b3b-fddd-414b-aff3-a8d971dc9a60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:21:29.503323 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.503210    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe657b3b-fddd-414b-aff3-a8d971dc9a60-kube-api-access-dzr5g" (OuterVolumeSpecName: "kube-api-access-dzr5g") pod "fe657b3b-fddd-414b-aff3-a8d971dc9a60" (UID: "fe657b3b-fddd-414b-aff3-a8d971dc9a60"). InnerVolumeSpecName "kube-api-access-dzr5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:21:29.503831 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.503785    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8498bb9-359d-4c21-b59b-f5eed6d923dd-kube-api-access-8k9dr" (OuterVolumeSpecName: "kube-api-access-8k9dr") pod "f8498bb9-359d-4c21-b59b-f5eed6d923dd" (UID: "f8498bb9-359d-4c21-b59b-f5eed6d923dd"). InnerVolumeSpecName "kube-api-access-8k9dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:21:29.503831 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.503799    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8498bb9-359d-4c21-b59b-f5eed6d923dd" containerID="782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768" exitCode=0
Jul 24 23:21:29.504016 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.503867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4" event={"ID":"f8498bb9-359d-4c21-b59b-f5eed6d923dd","Type":"ContainerDied","Data":"782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768"}
Jul 24 23:21:29.504016 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.503890    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"
Jul 24 23:21:29.504016 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.503959    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4" event={"ID":"f8498bb9-359d-4c21-b59b-f5eed6d923dd","Type":"ContainerDied","Data":"6f8967448766b469780018a1615337f990cb55a1f4deee52b715f94f63bd8202"}
Jul 24 23:21:29.504016 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.503995    2563 scope.go:117] "RemoveContainer" containerID="782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768"
Jul 24 23:21:29.506283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.506257    2563 generic.go:356] "Generic (PLEG): container finished" podID="159cae97-05f0-4465-ad1e-86ca0c78f9ec" containerID="86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9" exitCode=0
Jul 24 23:21:29.506829 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.506304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf" event={"ID":"159cae97-05f0-4465-ad1e-86ca0c78f9ec","Type":"ContainerDied","Data":"86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9"}
Jul 24 23:21:29.506829 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.506552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf" event={"ID":"159cae97-05f0-4465-ad1e-86ca0c78f9ec","Type":"ContainerDied","Data":"bd4513c310253398703a94605cd2df8c3ca76e2b1bfee0e465b77e88eeaabe71"}
Jul 24 23:21:29.506829 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.506332    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"
Jul 24 23:21:29.508452 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.508427    2563 generic.go:356] "Generic (PLEG): container finished" podID="fe657b3b-fddd-414b-aff3-a8d971dc9a60" containerID="9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc" exitCode=0
Jul 24 23:21:29.508605 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.508549    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"
Jul 24 23:21:29.508605 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.508588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc" event={"ID":"fe657b3b-fddd-414b-aff3-a8d971dc9a60","Type":"ContainerDied","Data":"9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc"}
Jul 24 23:21:29.508739 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.508629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc" event={"ID":"fe657b3b-fddd-414b-aff3-a8d971dc9a60","Type":"ContainerDied","Data":"8e8f632913fe76f4d9beb90c0b1a6247e41d1ad528cb90b4c8dcc00e382fae16"}
Jul 24 23:21:29.523871 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.523851    2563 scope.go:117] "RemoveContainer" containerID="782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768"
Jul 24 23:21:29.524439 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:21:29.524399    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768\": container with ID starting with 782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768 not found: ID does not exist" containerID="782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768"
Jul 24 23:21:29.524597 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.524450    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768"} err="failed to get container status \"782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768\": rpc error: code = NotFound desc = could not find container \"782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768\": container with ID starting with 782b02792bc902ab0682a9425fc5652cba2c30e84e1a2e10992efb4d280db768 not found: ID does not exist"
Jul 24 23:21:29.524597 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.524478    2563 scope.go:117] "RemoveContainer" containerID="86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9"
Jul 24 23:21:29.533617 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.533596    2563 scope.go:117] "RemoveContainer" containerID="86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9"
Jul 24 23:21:29.534007 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:21:29.533973    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9\": container with ID starting with 86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9 not found: ID does not exist" containerID="86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9"
Jul 24 23:21:29.534121 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.534015    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9"} err="failed to get container status \"86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9\": rpc error: code = NotFound desc = could not find container \"86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9\": container with ID starting with 86a1aa29f765a1f79763e0e137ad51b388023b5deb2f80ab25bf0bf5e6c241c9 not found: ID does not exist"
Jul 24 23:21:29.534121 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.534041    2563 scope.go:117] "RemoveContainer" containerID="9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc"
Jul 24 23:21:29.542328 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.542309    2563 scope.go:117] "RemoveContainer" containerID="9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc"
Jul 24 23:21:29.542665 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:21:29.542644    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc\": container with ID starting with 9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc not found: ID does not exist" containerID="9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc"
Jul 24 23:21:29.542747 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.542678    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc"} err="failed to get container status \"9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc\": rpc error: code = NotFound desc = could not find container \"9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc\": container with ID starting with 9c944c63aba6a71880a035414b96b20c94f8bf9fb9b6b01f5a96c691d83fb4fc not found: ID does not exist"
Jul 24 23:21:29.543847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.543827    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"]
Jul 24 23:21:29.547634 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.547609    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4813d775-c44ff6544-t55sc"]
Jul 24 23:21:29.557604 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.557574    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"]
Jul 24 23:21:29.561447 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.561421    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4813d775-c4fff6c6d-prbk4"]
Jul 24 23:21:29.571193 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.571163    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"]
Jul 24 23:21:29.573395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.573365    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4813d775-5ffcb95c94-hf9bf"]
Jul 24 23:21:29.598256 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.598219    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159cae97-05f0-4465-ad1e-86ca0c78f9ec-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:21:29.598256 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.598251    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzsk\" (UniqueName: \"kubernetes.io/projected/159cae97-05f0-4465-ad1e-86ca0c78f9ec-kube-api-access-6qzsk\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:21:29.598256 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.598263    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe657b3b-fddd-414b-aff3-a8d971dc9a60-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:21:29.598488 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.598273    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k9dr\" (UniqueName: \"kubernetes.io/projected/f8498bb9-359d-4c21-b59b-f5eed6d923dd-kube-api-access-8k9dr\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:21:29.598488 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.598287    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8498bb9-359d-4c21-b59b-f5eed6d923dd-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:21:29.598488 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.598299    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe657b3b-fddd-414b-aff3-a8d971dc9a60-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:21:29.598488 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.598307    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzr5g\" (UniqueName: \"kubernetes.io/projected/fe657b3b-fddd-414b-aff3-a8d971dc9a60-kube-api-access-dzr5g\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:21:29.842993 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.842962    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="159cae97-05f0-4465-ad1e-86ca0c78f9ec" path="/var/lib/kubelet/pods/159cae97-05f0-4465-ad1e-86ca0c78f9ec/volumes"
Jul 24 23:21:29.843328 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.843314    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8498bb9-359d-4c21-b59b-f5eed6d923dd" path="/var/lib/kubelet/pods/f8498bb9-359d-4c21-b59b-f5eed6d923dd/volumes"
Jul 24 23:21:29.843679 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:29.843666    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe657b3b-fddd-414b-aff3-a8d971dc9a60" path="/var/lib/kubelet/pods/fe657b3b-fddd-414b-aff3-a8d971dc9a60/volumes"
Jul 24 23:21:33.715821 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:33.715777    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"]
Jul 24 23:21:33.718340 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:33.718309    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4813d775-8vtcp"]
Jul 24 23:21:33.720111 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:33.720088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:21:33.841097 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:33.841055    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8fb8a6-dfe8-4a91-851a-5f6bce1245db" path="/var/lib/kubelet/pods/9a8fb8a6-dfe8-4a91-851a-5f6bce1245db/volumes"
Jul 24 23:21:36.908115 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:36.908069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:21:38.528962 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:38.528927    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:21:41.988461 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:21:41.988422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:18.457600 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:18.457560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:18.967266 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:18.967237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:24.024812 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:24.024772    2563 scope.go:117] "RemoveContainer" containerID="56db39cb409cfb04fb5be6361e7cfd86f7c0b8cd8ce8f4b05a89a37925220726"
Jul 24 23:22:24.033055 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:24.033036    2563 scope.go:117] "RemoveContainer" containerID="14c988e6bd9b03e2cbe07bee3ffa136f5412e9fa4f23766b0c0aada1f85f43f8"
Jul 24 23:22:24.384708 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:24.384672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:24.487415 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:24.487384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:24.681226 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:24.681101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:24.913977 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:24.913945    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:25.959861 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:25.959825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:26.058679 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:26.058642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:26.158917 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:26.158888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:26.877251 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:26.877213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:26.976006 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:26.975973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:27.076202 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:27.076170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:28.445548 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:28.445497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:28.681496 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:28.681461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:22:34.200728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:34.200697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:34.301404 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:34.301366    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:34.867763 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:34.867724    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:36.587641 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:36.587606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:36.635415 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:36.635377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:40.777739 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:40.777704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:40.836014 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:40.835977    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:22:40.843408 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:40.843375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:40.919452 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:40.919417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:41.019809 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:41.019731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:51.494339 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:51.494303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:22:51.833692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:22:51.833600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:12.834895 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:12.834845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:19.139416 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:19.139382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:19.240339 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:19.240299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:19.487095 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:19.487002    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:19.802266 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:19.802185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:20.037932 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:20.037901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:22.585766 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.585735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:22.684681 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.684647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:22.777233 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.777198    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"]
Jul 24 23:23:22.777774 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.777752    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe657b3b-fddd-414b-aff3-a8d971dc9a60" containerName="maas-api"
Jul 24 23:23:22.777948 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.777926    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe657b3b-fddd-414b-aff3-a8d971dc9a60" containerName="maas-api"
Jul 24 23:23:22.777948 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.777948    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="159cae97-05f0-4465-ad1e-86ca0c78f9ec" containerName="payload-processing"
Jul 24 23:23:22.778102 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.777959    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="159cae97-05f0-4465-ad1e-86ca0c78f9ec" containerName="payload-processing"
Jul 24 23:23:22.778102 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.777991    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8498bb9-359d-4c21-b59b-f5eed6d923dd" containerName="payload-pre-processing"
Jul 24 23:23:22.778102 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.777999    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8498bb9-359d-4c21-b59b-f5eed6d923dd" containerName="payload-pre-processing"
Jul 24 23:23:22.778102 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.778014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8fb8a6-dfe8-4a91-851a-5f6bce1245db" containerName="revoke-keys"
Jul 24 23:23:22.778102 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.778022    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8fb8a6-dfe8-4a91-851a-5f6bce1245db" containerName="revoke-keys"
Jul 24 23:23:22.778273 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.778111    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe657b3b-fddd-414b-aff3-a8d971dc9a60" containerName="maas-api"
Jul 24 23:23:22.778273 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.778123    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="159cae97-05f0-4465-ad1e-86ca0c78f9ec" containerName="payload-processing"
Jul 24 23:23:22.778273 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.778135    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8fb8a6-dfe8-4a91-851a-5f6bce1245db" containerName="revoke-keys"
Jul 24 23:23:22.778273 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.778145    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8498bb9-359d-4c21-b59b-f5eed6d923dd" containerName="payload-pre-processing"
Jul 24 23:23:22.781376 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.781360    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:22.788878 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.788854    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3b391a-serving-cert\""
Jul 24 23:23:22.788878 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.788866    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:23:22.790148 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.790131    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:23:22.790227 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.790185    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:23:22.797810 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.797787    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"]
Jul 24 23:23:22.823416 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.823381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:22.858789 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.858751    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzpc\" (UniqueName: \"kubernetes.io/projected/ed3d531e-9567-4ab5-909b-30124653b84f-kube-api-access-gqzpc\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:22.858789 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.858788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:22.859051 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.858894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d531e-9567-4ab5-909b-30124653b84f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:22.959846 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.959811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzpc\" (UniqueName: \"kubernetes.io/projected/ed3d531e-9567-4ab5-909b-30124653b84f-kube-api-access-gqzpc\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:22.959846 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.959854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:22.960127 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.959928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d531e-9567-4ab5-909b-30124653b84f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:22.960127 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:23:22.960033    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-serving-cert: secret "maas-api-e2e-inf-a-3b391a-serving-cert" not found
Jul 24 23:23:22.960127 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:23:22.960109    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls podName:ed3d531e-9567-4ab5-909b-30124653b84f nodeName:}" failed. No retries permitted until 2026-07-24 23:23:23.460091519 +0000 UTC m=+2220.093370437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls") pod "maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg" (UID: "ed3d531e-9567-4ab5-909b-30124653b84f") : secret "maas-api-e2e-inf-a-3b391a-serving-cert" not found
Jul 24 23:23:22.960639 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.960617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d531e-9567-4ab5-909b-30124653b84f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:22.970996 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:22.970973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzpc\" (UniqueName: \"kubernetes.io/projected/ed3d531e-9567-4ab5-909b-30124653b84f-kube-api-access-gqzpc\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:23.183354 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.183239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:23.292901 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.292862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:23.383407 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.383362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:23.384129 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.384107    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"]
Jul 24 23:23:23.387405 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.387390    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:23.390110 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.390086    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3b391a\""
Jul 24 23:23:23.398168 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.398144    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"]
Jul 24 23:23:23.430140 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.430108    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"]
Jul 24 23:23:23.433331 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.433315    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:23.435874 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.435817    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3b391a-dockercfg-v7rfv\""
Jul 24 23:23:23.442696 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.442672    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"]
Jul 24 23:23:23.464820 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.464787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77b8873-1154-4629-9bfb-def4cc981d29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6\" (UID: \"d77b8873-1154-4629-9bfb-def4cc981d29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:23.464990 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.464832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxl5\" (UniqueName: \"kubernetes.io/projected/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-kube-api-access-vtxl5\") pod \"payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw\" (UID: \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:23.464990 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.464941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw\" (UID: \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:23.465068 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.464989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68hz\" (UniqueName: \"kubernetes.io/projected/d77b8873-1154-4629-9bfb-def4cc981d29-kube-api-access-n68hz\") pod \"payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6\" (UID: \"d77b8873-1154-4629-9bfb-def4cc981d29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:23.465068 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.465044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:23.467439 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.467418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls\") pod \"maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:23.566033 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.565987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68hz\" (UniqueName: \"kubernetes.io/projected/d77b8873-1154-4629-9bfb-def4cc981d29-kube-api-access-n68hz\") pod \"payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6\" (UID: \"d77b8873-1154-4629-9bfb-def4cc981d29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:23.566255 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.566046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77b8873-1154-4629-9bfb-def4cc981d29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6\" (UID: \"d77b8873-1154-4629-9bfb-def4cc981d29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:23.566255 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.566073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxl5\" (UniqueName: \"kubernetes.io/projected/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-kube-api-access-vtxl5\") pod \"payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw\" (UID: \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:23.566255 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.566112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw\" (UID: \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:23.566839 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.566820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw\" (UID: \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:23.566924 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.566824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77b8873-1154-4629-9bfb-def4cc981d29-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6\" (UID: \"d77b8873-1154-4629-9bfb-def4cc981d29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:23.581974 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.581940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68hz\" (UniqueName: \"kubernetes.io/projected/d77b8873-1154-4629-9bfb-def4cc981d29-kube-api-access-n68hz\") pod \"payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6\" (UID: \"d77b8873-1154-4629-9bfb-def4cc981d29\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:23.582107 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.582005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxl5\" (UniqueName: \"kubernetes.io/projected/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-kube-api-access-vtxl5\") pod \"payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw\" (UID: \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:23.693219 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.693105    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:23.698091 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.698061    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:23.749055 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.749022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3b391a-dockercfg-v7rfv\""
Jul 24 23:23:23.756421 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.756194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:23.854486 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.854424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"]
Jul 24 23:23:23.924283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:23.924222    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"]
Jul 24 23:23:23.926383 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:23:23.926354    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77b8873_1154_4629_9bfb_def4cc981d29.slice/crio-6f19cf56e568c292736a45b3b9ff375d6a8c93f0b198aa2c09c44b4ca90d1a5c WatchSource:0}: Error finding container 6f19cf56e568c292736a45b3b9ff375d6a8c93f0b198aa2c09c44b4ca90d1a5c: Status 404 returned error can't find the container with id 6f19cf56e568c292736a45b3b9ff375d6a8c93f0b198aa2c09c44b4ca90d1a5c
Jul 24 23:23:24.065149 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:24.065102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6" event={"ID":"d77b8873-1154-4629-9bfb-def4cc981d29","Type":"ContainerStarted","Data":"c8dbfa33a89ac6b721f32268fe41cfd7e5078b5efa567faed202b39b4bbee2c3"}
Jul 24 23:23:24.065583 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:24.065169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6" event={"ID":"d77b8873-1154-4629-9bfb-def4cc981d29","Type":"ContainerStarted","Data":"6f19cf56e568c292736a45b3b9ff375d6a8c93f0b198aa2c09c44b4ca90d1a5c"}
Jul 24 23:23:24.065583 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:24.065220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:24.065726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:24.065605    2563 scope.go:117] "RemoveContainer" containerID="db44825c442ecacb28d1d67af1e669f1c5478806bb7046917e814983580f6379"
Jul 24 23:23:24.066665 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:24.066640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg" event={"ID":"ed3d531e-9567-4ab5-909b-30124653b84f","Type":"ContainerStarted","Data":"d37b902bec1cb810c89d282cbdc8f25f89d0693bada1462d7c03d608cfed355c"}
Jul 24 23:23:24.085089 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:24.085061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"]
Jul 24 23:23:24.085626 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:24.085575    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6" podStartSLOduration=1.085560144 podStartE2EDuration="1.085560144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:24.084637632 +0000 UTC m=+2220.717916575" watchObservedRunningTime="2026-07-24 23:23:24.085560144 +0000 UTC m=+2220.718839085"
Jul 24 23:23:24.087332 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:23:24.087308    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57fe93d_22c1_4d79_be1b_b76cf8ce88d1.slice/crio-6f8e8241cb4f4c431b60bd8019714b30a954fa400a56a3e6d155f3e78d4d466f WatchSource:0}: Error finding container 6f8e8241cb4f4c431b60bd8019714b30a954fa400a56a3e6d155f3e78d4d466f: Status 404 returned error can't find the container with id 6f8e8241cb4f4c431b60bd8019714b30a954fa400a56a3e6d155f3e78d4d466f
Jul 24 23:23:25.072603 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:25.072562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg" event={"ID":"ed3d531e-9567-4ab5-909b-30124653b84f","Type":"ContainerStarted","Data":"208c28b96b2331da0606508f0939923c8b3d00d55f73959cddee337d76bd69bc"}
Jul 24 23:23:25.073087 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:25.072629    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:25.074093 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:25.074068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw" event={"ID":"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1","Type":"ContainerStarted","Data":"a3a53dbdb328bbc4c94e1dcb6983bdbdce39550c50bd38a9e93797f8cad13c8c"}
Jul 24 23:23:25.074220 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:25.074116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw" event={"ID":"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1","Type":"ContainerStarted","Data":"6f8e8241cb4f4c431b60bd8019714b30a954fa400a56a3e6d155f3e78d4d466f"}
Jul 24 23:23:25.074277 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:25.074256    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:25.089336 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:25.089283    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg" podStartSLOduration=2.655750128 podStartE2EDuration="3.089265909s" totalImagesPullingTime="433.515781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:25.088561748 +0000 UTC m=+2221.721840686" watchObservedRunningTime="2026-07-24 23:23:25.089265909 +0000 UTC m=+2221.722544828"
Jul 24 23:23:25.103376 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:25.103324    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw" podStartSLOduration=2.103304995 podStartE2EDuration="2.103304995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:25.10274642 +0000 UTC m=+2221.736025395" watchObservedRunningTime="2026-07-24 23:23:25.103304995 +0000 UTC m=+2221.736583928"
Jul 24 23:23:27.072283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:27.072256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:27.457774 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:27.457738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:27.701617 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:27.701583    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:28.272795 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:28.272761    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:30.034902 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.034865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:23:30.607351 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.607308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:30.683671 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.683641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:30.688222 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.688193    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"]
Jul 24 23:23:30.691909 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.691886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.694595 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.694563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-313c6a-serving-cert\""
Jul 24 23:23:30.702935 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.702899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"]
Jul 24 23:23:30.731454 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.731416    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v945b\" (UniqueName: \"kubernetes.io/projected/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-kube-api-access-v945b\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.731626 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.731472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.731689 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.731631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.749911 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.749874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:30.833038 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.833003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v945b\" (UniqueName: \"kubernetes.io/projected/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-kube-api-access-v945b\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.833251 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.833051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.833251 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.833105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.833750 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.833732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.835630 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.835602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.840988 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.840957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v945b\" (UniqueName: \"kubernetes.io/projected/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-kube-api-access-v945b\") pod \"maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:30.852395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:30.852364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:31.003992 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.003897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:31.085076 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.085046    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:23:31.151164 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.151133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"]
Jul 24 23:23:31.154024 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:23:31.153992    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8325461d_3bc7_42bb_8d78_9eb5150cf4f2.slice/crio-2305fbb26c624e1dffc65580437e97bd8b2b713170202b8e5568039c99da3627 WatchSource:0}: Error finding container 2305fbb26c624e1dffc65580437e97bd8b2b713170202b8e5568039c99da3627: Status 404 returned error can't find the container with id 2305fbb26c624e1dffc65580437e97bd8b2b713170202b8e5568039c99da3627
Jul 24 23:23:31.374532 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.374492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:31.470384 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.470355    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"]
Jul 24 23:23:31.473462 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.473446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:31.476136 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.476107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-313c6a\""
Jul 24 23:23:31.476235 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.476172    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-313c6a-dockercfg-r4qtq\""
Jul 24 23:23:31.479379 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.479355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:31.483441 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.483373    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"]
Jul 24 23:23:31.513709 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.513682    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"]
Jul 24 23:23:31.516886 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.516870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:31.527335 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.527306    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"]
Jul 24 23:23:31.541156 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.541127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f141dce4-7492-44c2-a5a7-38cae63d03f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb\" (UID: \"f141dce4-7492-44c2-a5a7-38cae63d03f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:31.541315 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.541170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmmm\" (UniqueName: \"kubernetes.io/projected/f141dce4-7492-44c2-a5a7-38cae63d03f7-kube-api-access-smmmm\") pod \"payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb\" (UID: \"f141dce4-7492-44c2-a5a7-38cae63d03f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:31.641963 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.641853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmmm\" (UniqueName: \"kubernetes.io/projected/f141dce4-7492-44c2-a5a7-38cae63d03f7-kube-api-access-smmmm\") pod \"payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb\" (UID: \"f141dce4-7492-44c2-a5a7-38cae63d03f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:31.642137 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.641970    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7hh\" (UniqueName: \"kubernetes.io/projected/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-kube-api-access-lb7hh\") pod \"payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc\" (UID: \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:31.642137 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.642004    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc\" (UID: \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:31.642137 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.642037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f141dce4-7492-44c2-a5a7-38cae63d03f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb\" (UID: \"f141dce4-7492-44c2-a5a7-38cae63d03f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:31.642797 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.642772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f141dce4-7492-44c2-a5a7-38cae63d03f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb\" (UID: \"f141dce4-7492-44c2-a5a7-38cae63d03f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:31.650221 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.650197    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmmm\" (UniqueName: \"kubernetes.io/projected/f141dce4-7492-44c2-a5a7-38cae63d03f7-kube-api-access-smmmm\") pod \"payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb\" (UID: \"f141dce4-7492-44c2-a5a7-38cae63d03f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:31.742778 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.742741    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7hh\" (UniqueName: \"kubernetes.io/projected/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-kube-api-access-lb7hh\") pod \"payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc\" (UID: \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:31.742778 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.742782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc\" (UID: \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:31.743463 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.743446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc\" (UID: \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:31.750841 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.750813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7hh\" (UniqueName: \"kubernetes.io/projected/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-kube-api-access-lb7hh\") pod \"payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc\" (UID: \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:31.786691 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.786657    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:31.830013 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.829974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:31.929111 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.929080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"]
Jul 24 23:23:31.930766 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:23:31.930732    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf141dce4_7492_44c2_a5a7_38cae63d03f7.slice/crio-f21ee72dc7fe504ff54ef94c80e08fff6fdaa2de289727df3a357769a660b68f WatchSource:0}: Error finding container f21ee72dc7fe504ff54ef94c80e08fff6fdaa2de289727df3a357769a660b68f: Status 404 returned error can't find the container with id f21ee72dc7fe504ff54ef94c80e08fff6fdaa2de289727df3a357769a660b68f
Jul 24 23:23:31.992252 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:31.992218    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"]
Jul 24 23:23:31.994256 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:23:31.994226    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd353b2dc_e5ca_4675_a2da_9e31a04f64ab.slice/crio-6903cad0cfb4a718c33661cc10d671305ea6fd550e7bfb210208bce8c58da439 WatchSource:0}: Error finding container 6903cad0cfb4a718c33661cc10d671305ea6fd550e7bfb210208bce8c58da439: Status 404 returned error can't find the container with id 6903cad0cfb4a718c33661cc10d671305ea6fd550e7bfb210208bce8c58da439
Jul 24 23:23:32.115499 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.115461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p" event={"ID":"8325461d-3bc7-42bb-8d78-9eb5150cf4f2","Type":"ContainerStarted","Data":"d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436"}
Jul 24 23:23:32.116029 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.115541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p" event={"ID":"8325461d-3bc7-42bb-8d78-9eb5150cf4f2","Type":"ContainerStarted","Data":"2305fbb26c624e1dffc65580437e97bd8b2b713170202b8e5568039c99da3627"}
Jul 24 23:23:32.116029 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.115574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:32.120934 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.120889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc" event={"ID":"d353b2dc-e5ca-4675-a2da-9e31a04f64ab","Type":"ContainerStarted","Data":"9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364"}
Jul 24 23:23:32.121132 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.120950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc" event={"ID":"d353b2dc-e5ca-4675-a2da-9e31a04f64ab","Type":"ContainerStarted","Data":"6903cad0cfb4a718c33661cc10d671305ea6fd550e7bfb210208bce8c58da439"}
Jul 24 23:23:32.121132 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.120983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:32.122511 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.122471    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb" event={"ID":"f141dce4-7492-44c2-a5a7-38cae63d03f7","Type":"ContainerStarted","Data":"a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4"}
Jul 24 23:23:32.122633 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.122562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb" event={"ID":"f141dce4-7492-44c2-a5a7-38cae63d03f7","Type":"ContainerStarted","Data":"f21ee72dc7fe504ff54ef94c80e08fff6fdaa2de289727df3a357769a660b68f"}
Jul 24 23:23:32.122633 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.122599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:32.134352 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.134296    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p" podStartSLOduration=1.820888866 podStartE2EDuration="2.134278206s" totalImagesPullingTime="313.38934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:32.132828159 +0000 UTC m=+2228.766107103" watchObservedRunningTime="2026-07-24 23:23:32.134278206 +0000 UTC m=+2228.767557146"
Jul 24 23:23:32.149041 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.148783    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb" podStartSLOduration=1.148763637 podStartE2EDuration="1.148763637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:32.148386852 +0000 UTC m=+2228.781665807" watchObservedRunningTime="2026-07-24 23:23:32.148763637 +0000 UTC m=+2228.782042578"
Jul 24 23:23:32.164294 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:32.164242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc" podStartSLOduration=1.164225609 podStartE2EDuration="1.164225609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:32.162893284 +0000 UTC m=+2228.796172224" watchObservedRunningTime="2026-07-24 23:23:32.164225609 +0000 UTC m=+2228.797504549"
Jul 24 23:23:35.076376 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:35.076344    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:23:35.111631 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:35.111598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:36.081596 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:36.081564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:23:38.132843 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:38.132815    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:23:38.782396 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:38.782364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:23:43.130155 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:43.130123    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:23:43.130627 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:43.130201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:23:44.729032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:23:44.728994    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:24:04.093408 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:24:04.093299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:24:04.722536 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:24:04.722490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:24:08.028498 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:24:08.028457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:24:22.320984 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:24:22.320936    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:24:24.084700 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:24:24.084667    2563 scope.go:117] "RemoveContainer" containerID="6d2b2143b642d25e3afe7e130c0dc4bef6514f36e5f88a4ed1d1935f094554f6"
Jul 24 23:25:04.908330 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:04.908275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:06.399042 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:06.399003    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:09.623796 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:09.623754    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:25:15.997163 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:15.997045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:16.367633 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.367591    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"]
Jul 24 23:25:16.367913 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.367875    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg" podUID="ed3d531e-9567-4ab5-909b-30124653b84f" containerName="maas-api" containerID="cri-o://208c28b96b2331da0606508f0939923c8b3d00d55f73959cddee337d76bd69bc" gracePeriod=30
Jul 24 23:25:16.388870 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.388837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:16.420925 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.420882    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"]
Jul 24 23:25:16.421168 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.421142    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6" podUID="d77b8873-1154-4629-9bfb-def4cc981d29" containerName="payload-processing" containerID="cri-o://c8dbfa33a89ac6b721f32268fe41cfd7e5078b5efa567faed202b39b4bbee2c3" gracePeriod=30
Jul 24 23:25:16.428569 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.428506    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"]
Jul 24 23:25:16.428802 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.428777    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw" podUID="f57fe93d-22c1-4d79-be1b-b76cf8ce88d1" containerName="payload-pre-processing" containerID="cri-o://a3a53dbdb328bbc4c94e1dcb6983bdbdce39550c50bd38a9e93797f8cad13c8c" gracePeriod=30
Jul 24 23:25:16.459740 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.459714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:16.514779 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.514742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:16.640544 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.640498    2563 generic.go:356] "Generic (PLEG): container finished" podID="f57fe93d-22c1-4d79-be1b-b76cf8ce88d1" containerID="a3a53dbdb328bbc4c94e1dcb6983bdbdce39550c50bd38a9e93797f8cad13c8c" exitCode=0
Jul 24 23:25:16.640692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.640558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw" event={"ID":"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1","Type":"ContainerDied","Data":"a3a53dbdb328bbc4c94e1dcb6983bdbdce39550c50bd38a9e93797f8cad13c8c"}
Jul 24 23:25:16.642446 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.642420    2563 generic.go:356] "Generic (PLEG): container finished" podID="d77b8873-1154-4629-9bfb-def4cc981d29" containerID="c8dbfa33a89ac6b721f32268fe41cfd7e5078b5efa567faed202b39b4bbee2c3" exitCode=0
Jul 24 23:25:16.642582 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.642486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6" event={"ID":"d77b8873-1154-4629-9bfb-def4cc981d29","Type":"ContainerDied","Data":"c8dbfa33a89ac6b721f32268fe41cfd7e5078b5efa567faed202b39b4bbee2c3"}
Jul 24 23:25:16.644247 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.644221    2563 generic.go:356] "Generic (PLEG): container finished" podID="ed3d531e-9567-4ab5-909b-30124653b84f" containerID="208c28b96b2331da0606508f0939923c8b3d00d55f73959cddee337d76bd69bc" exitCode=0
Jul 24 23:25:16.644350 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.644264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg" event={"ID":"ed3d531e-9567-4ab5-909b-30124653b84f","Type":"ContainerDied","Data":"208c28b96b2331da0606508f0939923c8b3d00d55f73959cddee337d76bd69bc"}
Jul 24 23:25:16.686862 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.686828    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:25:16.722916 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.722883    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:25:16.732074 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.732048    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:25:16.790344 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790315    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqzpc\" (UniqueName: \"kubernetes.io/projected/ed3d531e-9567-4ab5-909b-30124653b84f-kube-api-access-gqzpc\") pod \"ed3d531e-9567-4ab5-909b-30124653b84f\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") "
Jul 24 23:25:16.790565 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790350    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtxl5\" (UniqueName: \"kubernetes.io/projected/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-kube-api-access-vtxl5\") pod \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\" (UID: \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\") "
Jul 24 23:25:16.790565 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77b8873-1154-4629-9bfb-def4cc981d29-plugins-config-volume\") pod \"d77b8873-1154-4629-9bfb-def4cc981d29\" (UID: \"d77b8873-1154-4629-9bfb-def4cc981d29\") "
Jul 24 23:25:16.790565 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790433    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d531e-9567-4ab5-909b-30124653b84f-openshift-service-ca\") pod \"ed3d531e-9567-4ab5-909b-30124653b84f\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") "
Jul 24 23:25:16.790565 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n68hz\" (UniqueName: \"kubernetes.io/projected/d77b8873-1154-4629-9bfb-def4cc981d29-kube-api-access-n68hz\") pod \"d77b8873-1154-4629-9bfb-def4cc981d29\" (UID: \"d77b8873-1154-4629-9bfb-def4cc981d29\") "
Jul 24 23:25:16.790565 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-plugins-config-volume\") pod \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\" (UID: \"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1\") "
Jul 24 23:25:16.790833 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790592    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls\") pod \"ed3d531e-9567-4ab5-909b-30124653b84f\" (UID: \"ed3d531e-9567-4ab5-909b-30124653b84f\") "
Jul 24 23:25:16.790893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3d531e-9567-4ab5-909b-30124653b84f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed3d531e-9567-4ab5-909b-30124653b84f" (UID: "ed3d531e-9567-4ab5-909b-30124653b84f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:25:16.790972 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.790944    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77b8873-1154-4629-9bfb-def4cc981d29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d77b8873-1154-4629-9bfb-def4cc981d29" (UID: "d77b8873-1154-4629-9bfb-def4cc981d29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:25:16.791036 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.791006    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f57fe93d-22c1-4d79-be1b-b76cf8ce88d1" (UID: "f57fe93d-22c1-4d79-be1b-b76cf8ce88d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:25:16.792642 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.792613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3d531e-9567-4ab5-909b-30124653b84f-kube-api-access-gqzpc" (OuterVolumeSpecName: "kube-api-access-gqzpc") pod "ed3d531e-9567-4ab5-909b-30124653b84f" (UID: "ed3d531e-9567-4ab5-909b-30124653b84f"). InnerVolumeSpecName "kube-api-access-gqzpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:25:16.792642 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.792624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-kube-api-access-vtxl5" (OuterVolumeSpecName: "kube-api-access-vtxl5") pod "f57fe93d-22c1-4d79-be1b-b76cf8ce88d1" (UID: "f57fe93d-22c1-4d79-be1b-b76cf8ce88d1"). InnerVolumeSpecName "kube-api-access-vtxl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:25:16.792814 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.792679    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77b8873-1154-4629-9bfb-def4cc981d29-kube-api-access-n68hz" (OuterVolumeSpecName: "kube-api-access-n68hz") pod "d77b8873-1154-4629-9bfb-def4cc981d29" (UID: "d77b8873-1154-4629-9bfb-def4cc981d29"). InnerVolumeSpecName "kube-api-access-n68hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:25:16.792862 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.792843    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed3d531e-9567-4ab5-909b-30124653b84f" (UID: "ed3d531e-9567-4ab5-909b-30124653b84f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:25:16.891922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.891828    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d77b8873-1154-4629-9bfb-def4cc981d29-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:16.891922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.891862    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3d531e-9567-4ab5-909b-30124653b84f-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:16.891922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.891873    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68hz\" (UniqueName: \"kubernetes.io/projected/d77b8873-1154-4629-9bfb-def4cc981d29-kube-api-access-n68hz\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:16.891922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.891883    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:16.891922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.891892    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3d531e-9567-4ab5-909b-30124653b84f-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:16.891922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.891900    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqzpc\" (UniqueName: \"kubernetes.io/projected/ed3d531e-9567-4ab5-909b-30124653b84f-kube-api-access-gqzpc\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:16.891922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:16.891908    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtxl5\" (UniqueName: \"kubernetes.io/projected/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1-kube-api-access-vtxl5\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:17.650023 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.649937    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6" event={"ID":"d77b8873-1154-4629-9bfb-def4cc981d29","Type":"ContainerDied","Data":"6f19cf56e568c292736a45b3b9ff375d6a8c93f0b198aa2c09c44b4ca90d1a5c"}
Jul 24 23:25:17.650023 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.649945    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"
Jul 24 23:25:17.650499 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.649997    2563 scope.go:117] "RemoveContainer" containerID="c8dbfa33a89ac6b721f32268fe41cfd7e5078b5efa567faed202b39b4bbee2c3"
Jul 24 23:25:17.651396 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.651317    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"
Jul 24 23:25:17.651396 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.651364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg" event={"ID":"ed3d531e-9567-4ab5-909b-30124653b84f","Type":"ContainerDied","Data":"d37b902bec1cb810c89d282cbdc8f25f89d0693bada1462d7c03d608cfed355c"}
Jul 24 23:25:17.652808 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.652784    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw" event={"ID":"f57fe93d-22c1-4d79-be1b-b76cf8ce88d1","Type":"ContainerDied","Data":"6f8e8241cb4f4c431b60bd8019714b30a954fa400a56a3e6d155f3e78d4d466f"}
Jul 24 23:25:17.652912 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.652812    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"
Jul 24 23:25:17.659395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.659369    2563 scope.go:117] "RemoveContainer" containerID="208c28b96b2331da0606508f0939923c8b3d00d55f73959cddee337d76bd69bc"
Jul 24 23:25:17.668794 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.668775    2563 scope.go:117] "RemoveContainer" containerID="a3a53dbdb328bbc4c94e1dcb6983bdbdce39550c50bd38a9e93797f8cad13c8c"
Jul 24 23:25:17.676193 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.676159    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"]
Jul 24 23:25:17.679200 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.679176    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3b391a-59cf986788-qhmwg"]
Jul 24 23:25:17.688991 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.688957    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"]
Jul 24 23:25:17.694722 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.694695    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3b391a-784fbc669b-4jdqw"]
Jul 24 23:25:17.705174 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.705142    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"]
Jul 24 23:25:17.706699 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.706675    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3b391a-6f6cfcfc5b-gfhw6"]
Jul 24 23:25:17.841641 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.841607    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d77b8873-1154-4629-9bfb-def4cc981d29" path="/var/lib/kubelet/pods/d77b8873-1154-4629-9bfb-def4cc981d29/volumes"
Jul 24 23:25:17.842083 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.842063    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed3d531e-9567-4ab5-909b-30124653b84f" path="/var/lib/kubelet/pods/ed3d531e-9567-4ab5-909b-30124653b84f/volumes"
Jul 24 23:25:17.842438 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:17.842422    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f57fe93d-22c1-4d79-be1b-b76cf8ce88d1" path="/var/lib/kubelet/pods/f57fe93d-22c1-4d79-be1b-b76cf8ce88d1/volumes"
Jul 24 23:25:20.978492 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:20.978455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:21.383507 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:21.383470    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:24.834925 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:24.834882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:25:25.514745 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:25.514704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:25.615904 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:25.615875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:27.303446 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.303407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:27.360937 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.360903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:27.565920 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.565834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:27.791688 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.791652    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:27.903697 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.903660    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"]
Jul 24 23:25:27.903999 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.903987    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f57fe93d-22c1-4d79-be1b-b76cf8ce88d1" containerName="payload-pre-processing"
Jul 24 23:25:27.904060 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.904000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f57fe93d-22c1-4d79-be1b-b76cf8ce88d1" containerName="payload-pre-processing"
Jul 24 23:25:27.904060 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.904014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3d531e-9567-4ab5-909b-30124653b84f" containerName="maas-api"
Jul 24 23:25:27.904060 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.904020    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3d531e-9567-4ab5-909b-30124653b84f" containerName="maas-api"
Jul 24 23:25:27.904060 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.904026    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77b8873-1154-4629-9bfb-def4cc981d29" containerName="payload-processing"
Jul 24 23:25:27.904060 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.904032    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77b8873-1154-4629-9bfb-def4cc981d29" containerName="payload-processing"
Jul 24 23:25:27.904207 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.904096    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77b8873-1154-4629-9bfb-def4cc981d29" containerName="payload-processing"
Jul 24 23:25:27.904207 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.904103    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f57fe93d-22c1-4d79-be1b-b76cf8ce88d1" containerName="payload-pre-processing"
Jul 24 23:25:27.904207 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.904112    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3d531e-9567-4ab5-909b-30124653b84f" containerName="maas-api"
Jul 24 23:25:27.908679 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.908653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"
Jul 24 23:25:27.911825 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.911794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:25:27.913435 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.913406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"]
Jul 24 23:25:27.990984 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:27.990933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f629cdd-9dfa-4b61-a4cb-2de9549c13c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-313c6a-krz27\" (UID: \"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"
Jul 24 23:25:28.091973 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:28.091937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f629cdd-9dfa-4b61-a4cb-2de9549c13c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-313c6a-krz27\" (UID: \"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"
Jul 24 23:25:28.092656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:28.092631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f629cdd-9dfa-4b61-a4cb-2de9549c13c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-313c6a-krz27\" (UID: \"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"
Jul 24 23:25:28.220901 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:28.220807    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"
Jul 24 23:25:28.363615 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:28.361227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"]
Jul 24 23:25:28.365597 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:25:28.365568    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f629cdd_9dfa_4b61_a4cb_2de9549c13c0.slice/crio-9112c9e7c69d076a45cfe9c1cbb6483395050241b350f2fa6561c2e642a75e9c WatchSource:0}: Error finding container 9112c9e7c69d076a45cfe9c1cbb6483395050241b350f2fa6561c2e642a75e9c: Status 404 returned error can't find the container with id 9112c9e7c69d076a45cfe9c1cbb6483395050241b350f2fa6561c2e642a75e9c
Jul 24 23:25:28.720782 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:28.720738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27" event={"ID":"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0","Type":"ContainerStarted","Data":"a81b979886beb6241c3c7fd365946327b366cce22bd80cd6cba651a58158919b"}
Jul 24 23:25:28.720963 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:28.720796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27" event={"ID":"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0","Type":"ContainerStarted","Data":"9112c9e7c69d076a45cfe9c1cbb6483395050241b350f2fa6561c2e642a75e9c"}
Jul 24 23:25:28.734821 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:28.734770    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27" podStartSLOduration=1.734756247 podStartE2EDuration="1.734756247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:25:28.734487496 +0000 UTC m=+2345.367766438" watchObservedRunningTime="2026-07-24 23:25:28.734756247 +0000 UTC m=+2345.368035187"
Jul 24 23:25:29.727073 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:29.727030    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f629cdd-9dfa-4b61-a4cb-2de9549c13c0" containerID="a81b979886beb6241c3c7fd365946327b366cce22bd80cd6cba651a58158919b" exitCode=0
Jul 24 23:25:29.727486 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:29.727121    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27" event={"ID":"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0","Type":"ContainerDied","Data":"a81b979886beb6241c3c7fd365946327b366cce22bd80cd6cba651a58158919b"}
Jul 24 23:25:30.871826 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:30.871788    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"
Jul 24 23:25:31.017619 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:31.017501    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f629cdd-9dfa-4b61-a4cb-2de9549c13c0-maas-api-service-ca\") pod \"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0\" (UID: \"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0\") "
Jul 24 23:25:31.017884 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:31.017858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f629cdd-9dfa-4b61-a4cb-2de9549c13c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f629cdd-9dfa-4b61-a4cb-2de9549c13c0" (UID: "0f629cdd-9dfa-4b61-a4cb-2de9549c13c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:25:31.118357 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:31.118320    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f629cdd-9dfa-4b61-a4cb-2de9549c13c0-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:31.737335 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:31.737304    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"
Jul 24 23:25:31.737503 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:31.737309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27" event={"ID":"0f629cdd-9dfa-4b61-a4cb-2de9549c13c0","Type":"ContainerDied","Data":"9112c9e7c69d076a45cfe9c1cbb6483395050241b350f2fa6561c2e642a75e9c"}
Jul 24 23:25:31.737503 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:31.737450    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9112c9e7c69d076a45cfe9c1cbb6483395050241b350f2fa6561c2e642a75e9c"
Jul 24 23:25:38.283116 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.283077    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"]
Jul 24 23:25:38.283635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.283374    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p" podUID="8325461d-3bc7-42bb-8d78-9eb5150cf4f2" containerName="maas-api" containerID="cri-o://d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436" gracePeriod=30
Jul 24 23:25:38.312752 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.312718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:38.336996 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.336961    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"]
Jul 24 23:25:38.337233 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.337188    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc" podUID="d353b2dc-e5ca-4675-a2da-9e31a04f64ab" containerName="payload-processing" containerID="cri-o://9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364" gracePeriod=30
Jul 24 23:25:38.345438 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.345404    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"]
Jul 24 23:25:38.345876 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.345847    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb" podUID="f141dce4-7492-44c2-a5a7-38cae63d03f7" containerName="payload-pre-processing" containerID="cri-o://a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4" gracePeriod=30
Jul 24 23:25:38.412762 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.412712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:38.513663 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.513629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:38.644013 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.643982    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:25:38.647389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.647352    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:25:38.741903 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.741877    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:25:38.772389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.772357    2563 generic.go:356] "Generic (PLEG): container finished" podID="d353b2dc-e5ca-4675-a2da-9e31a04f64ab" containerID="9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364" exitCode=0
Jul 24 23:25:38.772643 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.772416    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"
Jul 24 23:25:38.772643 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.772443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc" event={"ID":"d353b2dc-e5ca-4675-a2da-9e31a04f64ab","Type":"ContainerDied","Data":"9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364"}
Jul 24 23:25:38.772643 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.772534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc" event={"ID":"d353b2dc-e5ca-4675-a2da-9e31a04f64ab","Type":"ContainerDied","Data":"6903cad0cfb4a718c33661cc10d671305ea6fd550e7bfb210208bce8c58da439"}
Jul 24 23:25:38.772643 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.772605    2563 scope.go:117] "RemoveContainer" containerID="9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364"
Jul 24 23:25:38.773927 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.773907    2563 generic.go:356] "Generic (PLEG): container finished" podID="f141dce4-7492-44c2-a5a7-38cae63d03f7" containerID="a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4" exitCode=0
Jul 24 23:25:38.774044 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.773953    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"
Jul 24 23:25:38.774044 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.773992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb" event={"ID":"f141dce4-7492-44c2-a5a7-38cae63d03f7","Type":"ContainerDied","Data":"a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4"}
Jul 24 23:25:38.774162 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.774034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb" event={"ID":"f141dce4-7492-44c2-a5a7-38cae63d03f7","Type":"ContainerDied","Data":"f21ee72dc7fe504ff54ef94c80e08fff6fdaa2de289727df3a357769a660b68f"}
Jul 24 23:25:38.775406 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.775386    2563 generic.go:356] "Generic (PLEG): container finished" podID="8325461d-3bc7-42bb-8d78-9eb5150cf4f2" containerID="d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436" exitCode=0
Jul 24 23:25:38.775582 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.775430    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"
Jul 24 23:25:38.775582 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.775443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p" event={"ID":"8325461d-3bc7-42bb-8d78-9eb5150cf4f2","Type":"ContainerDied","Data":"d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436"}
Jul 24 23:25:38.775582 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.775483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p" event={"ID":"8325461d-3bc7-42bb-8d78-9eb5150cf4f2","Type":"ContainerDied","Data":"2305fbb26c624e1dffc65580437e97bd8b2b713170202b8e5568039c99da3627"}
Jul 24 23:25:38.782758 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.782734    2563 scope.go:117] "RemoveContainer" containerID="9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364"
Jul 24 23:25:38.783075 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:25:38.783046    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364\": container with ID starting with 9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364 not found: ID does not exist" containerID="9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364"
Jul 24 23:25:38.783159 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.783084    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364"} err="failed to get container status \"9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364\": rpc error: code = NotFound desc = could not find container \"9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364\": container with ID starting with 9a1993c03573c574a6791b5c90d78a0b25e2ee076290a66f24439901852e8364 not found: ID does not exist"
Jul 24 23:25:38.783159 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.783105    2563 scope.go:117] "RemoveContainer" containerID="a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4"
Jul 24 23:25:38.785390 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785356    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-maas-api-tls\") pod \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") "
Jul 24 23:25:38.785479 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785408    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb7hh\" (UniqueName: \"kubernetes.io/projected/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-kube-api-access-lb7hh\") pod \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\" (UID: \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\") "
Jul 24 23:25:38.785479 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f141dce4-7492-44c2-a5a7-38cae63d03f7-plugins-config-volume\") pod \"f141dce4-7492-44c2-a5a7-38cae63d03f7\" (UID: \"f141dce4-7492-44c2-a5a7-38cae63d03f7\") "
Jul 24 23:25:38.785604 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785502    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-openshift-service-ca\") pod \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") "
Jul 24 23:25:38.785658 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v945b\" (UniqueName: \"kubernetes.io/projected/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-kube-api-access-v945b\") pod \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\" (UID: \"8325461d-3bc7-42bb-8d78-9eb5150cf4f2\") "
Jul 24 23:25:38.785658 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785627    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smmmm\" (UniqueName: \"kubernetes.io/projected/f141dce4-7492-44c2-a5a7-38cae63d03f7-kube-api-access-smmmm\") pod \"f141dce4-7492-44c2-a5a7-38cae63d03f7\" (UID: \"f141dce4-7492-44c2-a5a7-38cae63d03f7\") "
Jul 24 23:25:38.785759 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785658    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-plugins-config-volume\") pod \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\" (UID: \"d353b2dc-e5ca-4675-a2da-9e31a04f64ab\") "
Jul 24 23:25:38.785985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785826    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8325461d-3bc7-42bb-8d78-9eb5150cf4f2" (UID: "8325461d-3bc7-42bb-8d78-9eb5150cf4f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:25:38.785985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f141dce4-7492-44c2-a5a7-38cae63d03f7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f141dce4-7492-44c2-a5a7-38cae63d03f7" (UID: "f141dce4-7492-44c2-a5a7-38cae63d03f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:25:38.785985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785942    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f141dce4-7492-44c2-a5a7-38cae63d03f7-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:38.785985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.785963    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:38.786366 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.786209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d353b2dc-e5ca-4675-a2da-9e31a04f64ab" (UID: "d353b2dc-e5ca-4675-a2da-9e31a04f64ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:25:38.787667 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.787645    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8325461d-3bc7-42bb-8d78-9eb5150cf4f2" (UID: "8325461d-3bc7-42bb-8d78-9eb5150cf4f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:25:38.787754 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.787732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-kube-api-access-v945b" (OuterVolumeSpecName: "kube-api-access-v945b") pod "8325461d-3bc7-42bb-8d78-9eb5150cf4f2" (UID: "8325461d-3bc7-42bb-8d78-9eb5150cf4f2"). InnerVolumeSpecName "kube-api-access-v945b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:25:38.787982 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.787960    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-kube-api-access-lb7hh" (OuterVolumeSpecName: "kube-api-access-lb7hh") pod "d353b2dc-e5ca-4675-a2da-9e31a04f64ab" (UID: "d353b2dc-e5ca-4675-a2da-9e31a04f64ab"). InnerVolumeSpecName "kube-api-access-lb7hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:25:38.788060 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.788028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f141dce4-7492-44c2-a5a7-38cae63d03f7-kube-api-access-smmmm" (OuterVolumeSpecName: "kube-api-access-smmmm") pod "f141dce4-7492-44c2-a5a7-38cae63d03f7" (UID: "f141dce4-7492-44c2-a5a7-38cae63d03f7"). InnerVolumeSpecName "kube-api-access-smmmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:25:38.791988 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.791965    2563 scope.go:117] "RemoveContainer" containerID="a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4"
Jul 24 23:25:38.792289 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:25:38.792270    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4\": container with ID starting with a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4 not found: ID does not exist" containerID="a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4"
Jul 24 23:25:38.792343 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.792297    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4"} err="failed to get container status \"a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4\": rpc error: code = NotFound desc = could not find container \"a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4\": container with ID starting with a58eec0eaa7c914962618b06ecb4284dbe8d315392909214fa80f413c73813c4 not found: ID does not exist"
Jul 24 23:25:38.792343 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.792315    2563 scope.go:117] "RemoveContainer" containerID="d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436"
Jul 24 23:25:38.801035 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.801015    2563 scope.go:117] "RemoveContainer" containerID="d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436"
Jul 24 23:25:38.801352 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:25:38.801320    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436\": container with ID starting with d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436 not found: ID does not exist" containerID="d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436"
Jul 24 23:25:38.801424 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.801366    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436"} err="failed to get container status \"d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436\": rpc error: code = NotFound desc = could not find container \"d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436\": container with ID starting with d82ffdd96227e289b3d5acac5a2e4e89b503d185f22dbb54673cec50ccbdd436 not found: ID does not exist"
Jul 24 23:25:38.887078 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.887036    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v945b\" (UniqueName: \"kubernetes.io/projected/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-kube-api-access-v945b\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:38.887078 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.887072    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smmmm\" (UniqueName: \"kubernetes.io/projected/f141dce4-7492-44c2-a5a7-38cae63d03f7-kube-api-access-smmmm\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:38.887078 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.887082    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:38.887333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.887093    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8325461d-3bc7-42bb-8d78-9eb5150cf4f2-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:38.887333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:38.887103    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb7hh\" (UniqueName: \"kubernetes.io/projected/d353b2dc-e5ca-4675-a2da-9e31a04f64ab-kube-api-access-lb7hh\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:25:39.099404 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.099362    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"]
Jul 24 23:25:39.102751 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.102720    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-313c6a-57466855db-8t5xc"]
Jul 24 23:25:39.112759 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.112729    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"]
Jul 24 23:25:39.118454 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.118418    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-313c6a-56649dbd4c-rdv9p"]
Jul 24 23:25:39.129785 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.129747    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"]
Jul 24 23:25:39.135573 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.132979    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-313c6a-6f96f49c44-wwqbb"]
Jul 24 23:25:39.841152 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.841122    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8325461d-3bc7-42bb-8d78-9eb5150cf4f2" path="/var/lib/kubelet/pods/8325461d-3bc7-42bb-8d78-9eb5150cf4f2/volumes"
Jul 24 23:25:39.841697 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.841675    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d353b2dc-e5ca-4675-a2da-9e31a04f64ab" path="/var/lib/kubelet/pods/d353b2dc-e5ca-4675-a2da-9e31a04f64ab/volumes"
Jul 24 23:25:39.842063 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:39.842045    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f141dce4-7492-44c2-a5a7-38cae63d03f7" path="/var/lib/kubelet/pods/f141dce4-7492-44c2-a5a7-38cae63d03f7/volumes"
Jul 24 23:25:41.230202 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:41.230162    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:25:42.977624 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:42.977593    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"]
Jul 24 23:25:42.978068 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:42.977928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:42.979131 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:42.979110    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-313c6a-krz27"]
Jul 24 23:25:43.352792 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:43.352760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:43.841585 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:43.841558    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f629cdd-9dfa-4b61-a4cb-2de9549c13c0" path="/var/lib/kubelet/pods/0f629cdd-9dfa-4b61-a4cb-2de9549c13c0/volumes"
Jul 24 23:25:49.009944 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:49.009896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:49.112110 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:49.112076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:49.272183 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:49.272095    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:49.493159 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:49.493128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:49.817818 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:49.817786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:50.133188 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:50.133159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:50.233285 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:50.233253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:53.425164 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.425132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:53.553689 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.553653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:53.649411 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649378    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"]
Jul 24 23:25:53.649726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649712    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f141dce4-7492-44c2-a5a7-38cae63d03f7" containerName="payload-pre-processing"
Jul 24 23:25:53.649726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649726    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f141dce4-7492-44c2-a5a7-38cae63d03f7" containerName="payload-pre-processing"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649735    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8325461d-3bc7-42bb-8d78-9eb5150cf4f2" containerName="maas-api"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649741    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8325461d-3bc7-42bb-8d78-9eb5150cf4f2" containerName="maas-api"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649748    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d353b2dc-e5ca-4675-a2da-9e31a04f64ab" containerName="payload-processing"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649754    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d353b2dc-e5ca-4675-a2da-9e31a04f64ab" containerName="payload-processing"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649765    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f629cdd-9dfa-4b61-a4cb-2de9549c13c0" containerName="revoke-keys"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649770    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f629cdd-9dfa-4b61-a4cb-2de9549c13c0" containerName="revoke-keys"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649827    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8325461d-3bc7-42bb-8d78-9eb5150cf4f2" containerName="maas-api"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649841    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f629cdd-9dfa-4b61-a4cb-2de9549c13c0" containerName="revoke-keys"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649849    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d353b2dc-e5ca-4675-a2da-9e31a04f64ab" containerName="payload-processing"
Jul 24 23:25:53.649859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.649857    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f141dce4-7492-44c2-a5a7-38cae63d03f7" containerName="payload-pre-processing"
Jul 24 23:25:53.654661 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.654638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:53.657382 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.657348    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4ab9b5-serving-cert\""
Jul 24 23:25:53.657609 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.657581    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:25:53.657609 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.657385    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:25:53.657797 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.657358    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:25:53.664626 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.664587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"]
Jul 24 23:25:53.683121 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.683037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:53.711695 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.711657    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:53.711883 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.711793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtz5\" (UniqueName: \"kubernetes.io/projected/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-kube-api-access-smtz5\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:53.711883 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.711840    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:53.813254 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.813216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtz5\" (UniqueName: \"kubernetes.io/projected/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-kube-api-access-smtz5\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:53.813254 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.813261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:53.813502 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.813291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:53.813502 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:25:53.813394    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-serving-cert: secret "maas-api-e2e-api-a-4ab9b5-serving-cert" not found
Jul 24 23:25:53.813502 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:25:53.813468    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls podName:d82457d9-1ec0-4dff-a6c0-e9d4d114e21c nodeName:}" failed. No retries permitted until 2026-07-24 23:25:54.31344453 +0000 UTC m=+2370.946723464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls") pod "maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts" (UID: "d82457d9-1ec0-4dff-a6c0-e9d4d114e21c") : secret "maas-api-e2e-api-a-4ab9b5-serving-cert" not found
Jul 24 23:25:53.814043 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.814024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:53.824097 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:53.824065    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtz5\" (UniqueName: \"kubernetes.io/projected/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-kube-api-access-smtz5\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:54.169782 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.169753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:54.317637 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.317602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:54.320017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.319996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls\") pod \"maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:54.332797 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.332763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:54.426738 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.426658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:54.427617 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.427595    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"]
Jul 24 23:25:54.430922 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.430903    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:54.433661 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.433628    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4ab9b5\""
Jul 24 23:25:54.433989 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.433630    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4ab9b5-dockercfg-wq8d9\""
Jul 24 23:25:54.441156 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.441130    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"]
Jul 24 23:25:54.478370 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.478331    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"]
Jul 24 23:25:54.481613 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.481594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:54.491482 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.491451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"]
Jul 24 23:25:54.519558 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.519478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvnjw\" (UniqueName: \"kubernetes.io/projected/8cabd655-bbcb-45e7-ad7d-d574e41ab860-kube-api-access-mvnjw\") pod \"payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh\" (UID: \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:54.519558 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.519557    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cabd655-bbcb-45e7-ad7d-d574e41ab860-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh\" (UID: \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:54.519783 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.519589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd\" (UID: \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:54.519783 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.519618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqmx\" (UniqueName: \"kubernetes.io/projected/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-kube-api-access-zwqmx\") pod \"payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd\" (UID: \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:54.566331 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.566288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:54.621016 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.620967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvnjw\" (UniqueName: \"kubernetes.io/projected/8cabd655-bbcb-45e7-ad7d-d574e41ab860-kube-api-access-mvnjw\") pod \"payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh\" (UID: \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:54.621132 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.621057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cabd655-bbcb-45e7-ad7d-d574e41ab860-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh\" (UID: \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:54.621132 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.621116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd\" (UID: \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:54.621208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.621164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqmx\" (UniqueName: \"kubernetes.io/projected/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-kube-api-access-zwqmx\") pod \"payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd\" (UID: \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:54.622085 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.622056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cabd655-bbcb-45e7-ad7d-d574e41ab860-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh\" (UID: \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:54.622351 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.622324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd\" (UID: \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:54.637179 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.637145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqmx\" (UniqueName: \"kubernetes.io/projected/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-kube-api-access-zwqmx\") pod \"payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd\" (UID: \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\") " pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:54.638124 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.638096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvnjw\" (UniqueName: \"kubernetes.io/projected/8cabd655-bbcb-45e7-ad7d-d574e41ab860-kube-api-access-mvnjw\") pod \"payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh\" (UID: \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:54.708552 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.708493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"]
Jul 24 23:25:54.710759 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:25:54.710726    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82457d9_1ec0_4dff_a6c0_e9d4d114e21c.slice/crio-c74697b5844ef46f097314780aa0b3cb4912b19acfc6a6467b73b5b4e3ed88db WatchSource:0}: Error finding container c74697b5844ef46f097314780aa0b3cb4912b19acfc6a6467b73b5b4e3ed88db: Status 404 returned error can't find the container with id c74697b5844ef46f097314780aa0b3cb4912b19acfc6a6467b73b5b4e3ed88db
Jul 24 23:25:54.712220 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.712192    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 23:25:54.744273 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.744238    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:54.792864 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.792829    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:54.866976 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.866417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts" event={"ID":"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c","Type":"ContainerStarted","Data":"c74697b5844ef46f097314780aa0b3cb4912b19acfc6a6467b73b5b4e3ed88db"}
Jul 24 23:25:54.889709 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.889682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"]
Jul 24 23:25:54.892871 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:25:54.892839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cabd655_bbcb_45e7_ad7d_d574e41ab860.slice/crio-68cd1e40489f2be792b8431c4391d7d1cca29972289ccc71413cb3360b91cb9c WatchSource:0}: Error finding container 68cd1e40489f2be792b8431c4391d7d1cca29972289ccc71413cb3360b91cb9c: Status 404 returned error can't find the container with id 68cd1e40489f2be792b8431c4391d7d1cca29972289ccc71413cb3360b91cb9c
Jul 24 23:25:54.949299 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:54.949272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"]
Jul 24 23:25:54.951715 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:25:54.951676    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87cf9a2_7062_45e5_b5d5_fee3f0627d8d.slice/crio-9917ba6cc29b001f2d0b8a83056dcfd3f2afd843051733261cc0976376580471 WatchSource:0}: Error finding container 9917ba6cc29b001f2d0b8a83056dcfd3f2afd843051733261cc0976376580471: Status 404 returned error can't find the container with id 9917ba6cc29b001f2d0b8a83056dcfd3f2afd843051733261cc0976376580471
Jul 24 23:25:55.873570 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.873529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd" event={"ID":"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d","Type":"ContainerStarted","Data":"4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80"}
Jul 24 23:25:55.874037 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.873605    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:25:55.874037 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.873628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd" event={"ID":"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d","Type":"ContainerStarted","Data":"9917ba6cc29b001f2d0b8a83056dcfd3f2afd843051733261cc0976376580471"}
Jul 24 23:25:55.875017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.874991    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts" event={"ID":"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c","Type":"ContainerStarted","Data":"ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a"}
Jul 24 23:25:55.875108 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.875047    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:25:55.876705 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.876674    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh" event={"ID":"8cabd655-bbcb-45e7-ad7d-d574e41ab860","Type":"ContainerStarted","Data":"bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301"}
Jul 24 23:25:55.876826 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.876721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh" event={"ID":"8cabd655-bbcb-45e7-ad7d-d574e41ab860","Type":"ContainerStarted","Data":"68cd1e40489f2be792b8431c4391d7d1cca29972289ccc71413cb3360b91cb9c"}
Jul 24 23:25:55.876826 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.876769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:25:55.888594 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.888544    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd" podStartSLOduration=1.8885086279999999 podStartE2EDuration="1.888508628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:25:55.888418778 +0000 UTC m=+2372.521697721" watchObservedRunningTime="2026-07-24 23:25:55.888508628 +0000 UTC m=+2372.521787562"
Jul 24 23:25:55.904798 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.904739    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts" podStartSLOduration=2.457832422 podStartE2EDuration="2.904720533s" totalImagesPullingTime="446.888111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:25:55.903548961 +0000 UTC m=+2372.536827893" watchObservedRunningTime="2026-07-24 23:25:55.904720533 +0000 UTC m=+2372.537999474"
Jul 24 23:25:55.918515 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:55.918457    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh" podStartSLOduration=1.918440428 podStartE2EDuration="1.918440428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:25:55.917577244 +0000 UTC m=+2372.550856184" watchObservedRunningTime="2026-07-24 23:25:55.918440428 +0000 UTC m=+2372.551719367"
Jul 24 23:25:57.925963 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:57.925927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:58.369373 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:58.369321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:58.680755 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:58.680658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:25:58.960085 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:25:58.959994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:00.226102 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:00.226059    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:26:01.556430 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.556393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:01.633882 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.633843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:01.639349 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.639311    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"]
Jul 24 23:26:01.642835 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.642811    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.645624 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.645601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-569db3-serving-cert\""
Jul 24 23:26:01.652745 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.652712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"]
Jul 24 23:26:01.703200 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.703170    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:01.790751 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.790708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szdcg\" (UniqueName: \"kubernetes.io/projected/25e76a50-a90f-4251-b4b6-786cdda47423-kube-api-access-szdcg\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.790914 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.790789    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25e76a50-a90f-4251-b4b6-786cdda47423-maas-api-tls\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.790914 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.790883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e76a50-a90f-4251-b4b6-786cdda47423-openshift-service-ca\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.887747 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.887720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:26:01.891143 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.891118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:01.891298 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.891216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e76a50-a90f-4251-b4b6-786cdda47423-openshift-service-ca\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.891298 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.891242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szdcg\" (UniqueName: \"kubernetes.io/projected/25e76a50-a90f-4251-b4b6-786cdda47423-kube-api-access-szdcg\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.891298 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.891278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25e76a50-a90f-4251-b4b6-786cdda47423-maas-api-tls\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.892068 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.892044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e76a50-a90f-4251-b4b6-786cdda47423-openshift-service-ca\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.893852 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.893824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25e76a50-a90f-4251-b4b6-786cdda47423-maas-api-tls\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.900414 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.900389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szdcg\" (UniqueName: \"kubernetes.io/projected/25e76a50-a90f-4251-b4b6-786cdda47423-kube-api-access-szdcg\") pod \"maas-api-e2e-api-b-569db3-788889b5b4-hfhxt\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:01.955962 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:01.955924    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:02.100578 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.100544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"]
Jul 24 23:26:02.102623 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:26:02.102593    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e76a50_a90f_4251_b4b6_786cdda47423.slice/crio-bd3fe3e9f44918ab1d38efc8970e6af142d110326c9c026bc8dffe9acc9d8df0 WatchSource:0}: Error finding container bd3fe3e9f44918ab1d38efc8970e6af142d110326c9c026bc8dffe9acc9d8df0: Status 404 returned error can't find the container with id bd3fe3e9f44918ab1d38efc8970e6af142d110326c9c026bc8dffe9acc9d8df0
Jul 24 23:26:02.237804 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.237718    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:02.333881 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.333850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:02.340793 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.340762    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"]
Jul 24 23:26:02.345179 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.345154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.348021 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.347991    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-569db3\""
Jul 24 23:26:02.348164 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.348022    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-569db3-dockercfg-8rwlf\""
Jul 24 23:26:02.354027 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.354004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"]
Jul 24 23:26:02.366453 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.366421    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"]
Jul 24 23:26:02.369793 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.369773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.381474 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.381446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"]
Jul 24 23:26:02.497457 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.497354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6278e82-02ac-415c-a54b-92e45a425cf9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx\" (UID: \"b6278e82-02ac-415c-a54b-92e45a425cf9\") " pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.497618 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.497469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvptb\" (UniqueName: \"kubernetes.io/projected/b6278e82-02ac-415c-a54b-92e45a425cf9-kube-api-access-dvptb\") pod \"payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx\" (UID: \"b6278e82-02ac-415c-a54b-92e45a425cf9\") " pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.497618 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.497578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/4ca0fefa-47fc-483d-950c-33c262d61a9e-kube-api-access-wtt6q\") pod \"payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9\" (UID: \"4ca0fefa-47fc-483d-950c-33c262d61a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.497743 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.497625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca0fefa-47fc-483d-950c-33c262d61a9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9\" (UID: \"4ca0fefa-47fc-483d-950c-33c262d61a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.598701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.598661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/4ca0fefa-47fc-483d-950c-33c262d61a9e-kube-api-access-wtt6q\") pod \"payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9\" (UID: \"4ca0fefa-47fc-483d-950c-33c262d61a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.599181 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.598723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca0fefa-47fc-483d-950c-33c262d61a9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9\" (UID: \"4ca0fefa-47fc-483d-950c-33c262d61a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.599181 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.598761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6278e82-02ac-415c-a54b-92e45a425cf9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx\" (UID: \"b6278e82-02ac-415c-a54b-92e45a425cf9\") " pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.599181 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.598855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvptb\" (UniqueName: \"kubernetes.io/projected/b6278e82-02ac-415c-a54b-92e45a425cf9-kube-api-access-dvptb\") pod \"payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx\" (UID: \"b6278e82-02ac-415c-a54b-92e45a425cf9\") " pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.599819 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.599786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca0fefa-47fc-483d-950c-33c262d61a9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9\" (UID: \"4ca0fefa-47fc-483d-950c-33c262d61a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.599932 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.599878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6278e82-02ac-415c-a54b-92e45a425cf9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx\" (UID: \"b6278e82-02ac-415c-a54b-92e45a425cf9\") " pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.607038 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.607003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/4ca0fefa-47fc-483d-950c-33c262d61a9e-kube-api-access-wtt6q\") pod \"payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9\" (UID: \"4ca0fefa-47fc-483d-950c-33c262d61a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.607156 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.607014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvptb\" (UniqueName: \"kubernetes.io/projected/b6278e82-02ac-415c-a54b-92e45a425cf9-kube-api-access-dvptb\") pod \"payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx\" (UID: \"b6278e82-02ac-415c-a54b-92e45a425cf9\") " pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.657627 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.657582    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.680965 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.680929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.813869 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.813835    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"]
Jul 24 23:26:02.816260 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:26:02.816219    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca0fefa_47fc_483d_950c_33c262d61a9e.slice/crio-81d729b3ffd10280e30017fc6c6f0163441c6d892e3b65be32c7f6fc430e914a WatchSource:0}: Error finding container 81d729b3ffd10280e30017fc6c6f0163441c6d892e3b65be32c7f6fc430e914a: Status 404 returned error can't find the container with id 81d729b3ffd10280e30017fc6c6f0163441c6d892e3b65be32c7f6fc430e914a
Jul 24 23:26:02.844980 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.844907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"]
Jul 24 23:26:02.847309 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:26:02.847277    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6278e82_02ac_415c_a54b_92e45a425cf9.slice/crio-5c7fb25494086c614909cd22694eda54943f45b96b42d5de473453c7c0c03537 WatchSource:0}: Error finding container 5c7fb25494086c614909cd22694eda54943f45b96b42d5de473453c7c0c03537: Status 404 returned error can't find the container with id 5c7fb25494086c614909cd22694eda54943f45b96b42d5de473453c7c0c03537
Jul 24 23:26:02.914343 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.914306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9" event={"ID":"4ca0fefa-47fc-483d-950c-33c262d61a9e","Type":"ContainerStarted","Data":"0b11ab1bef90ee0ae45c7798f78ce7e847b1c0ebe76c81c514a905ee308c0e06"}
Jul 24 23:26:02.914558 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.914375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9" event={"ID":"4ca0fefa-47fc-483d-950c-33c262d61a9e","Type":"ContainerStarted","Data":"81d729b3ffd10280e30017fc6c6f0163441c6d892e3b65be32c7f6fc430e914a"}
Jul 24 23:26:02.914558 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.914551    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:02.916242 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.916209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt" event={"ID":"25e76a50-a90f-4251-b4b6-786cdda47423","Type":"ContainerStarted","Data":"42e204b02715beae6a1869dcdc5976455f575b83f35f2d93f9bb5a28c817500e"}
Jul 24 23:26:02.916405 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.916262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt" event={"ID":"25e76a50-a90f-4251-b4b6-786cdda47423","Type":"ContainerStarted","Data":"bd3fe3e9f44918ab1d38efc8970e6af142d110326c9c026bc8dffe9acc9d8df0"}
Jul 24 23:26:02.916405 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.916366    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:02.917910 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.917877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx" event={"ID":"b6278e82-02ac-415c-a54b-92e45a425cf9","Type":"ContainerStarted","Data":"fd95e357e764041b8e0d8bc067dd6f99374665b5870d268f504a03a27fbca22a"}
Jul 24 23:26:02.918045 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.917916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx" event={"ID":"b6278e82-02ac-415c-a54b-92e45a425cf9","Type":"ContainerStarted","Data":"5c7fb25494086c614909cd22694eda54943f45b96b42d5de473453c7c0c03537"}
Jul 24 23:26:02.918045 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.917938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:02.933042 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.932989    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9" podStartSLOduration=0.93296955 podStartE2EDuration="932.96955ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:26:02.930785757 +0000 UTC m=+2379.564064702" watchObservedRunningTime="2026-07-24 23:26:02.93296955 +0000 UTC m=+2379.566248490"
Jul 24 23:26:02.949172 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.949112    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx" podStartSLOduration=0.949094539 podStartE2EDuration="949.094539ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:26:02.94734731 +0000 UTC m=+2379.580626252" watchObservedRunningTime="2026-07-24 23:26:02.949094539 +0000 UTC m=+2379.582373483"
Jul 24 23:26:02.964053 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:02.963994    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt" podStartSLOduration=1.6651943070000002 podStartE2EDuration="1.963971757s" totalImagesPullingTime="298.77745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:26:02.963326076 +0000 UTC m=+2379.596605032" watchObservedRunningTime="2026-07-24 23:26:02.963971757 +0000 UTC m=+2379.597250698"
Jul 24 23:26:05.966785 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:05.966740    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:06.884151 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:06.884116    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:26:06.884346 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:06.884200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:26:08.928593 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:08.928564    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:13.925558 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:13.925509    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:13.925981 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:13.925608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:15.135898 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:15.135862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:26:23.916841 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:23.916812    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:26:23.920072 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:23.920050    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:26:24.122496 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:24.122465    2563 scope.go:117] "RemoveContainer" containerID="54777d87e9f877f798b348c1a58f08cfb27343446a8aa194cc665691d45b0649"
Jul 24 23:26:29.636208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.636160    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"]
Jul 24 23:26:29.636721 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.636419    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts" podUID="d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" containerName="maas-api" containerID="cri-o://ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a" gracePeriod=30
Jul 24 23:26:29.663396 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.663367    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:29.688253 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.688218    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"]
Jul 24 23:26:29.688512 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.688482    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd" podUID="d87cf9a2-7062-45e5-b5d5-fee3f0627d8d" containerName="payload-processing" containerID="cri-o://4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80" gracePeriod=30
Jul 24 23:26:29.692962 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.692934    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"]
Jul 24 23:26:29.693194 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.693152    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh" podUID="8cabd655-bbcb-45e7-ad7d-d574e41ab860" containerName="payload-pre-processing" containerID="cri-o://bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301" gracePeriod=30
Jul 24 23:26:29.725497 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.725466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:29.783717 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.783671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:29.954454 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.954428    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:26:29.967784 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:29.967758    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:26:30.017956 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.017930    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:26:30.039692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.039663    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls\") pod \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") "
Jul 24 23:26:30.039885 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.039702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smtz5\" (UniqueName: \"kubernetes.io/projected/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-kube-api-access-smtz5\") pod \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") "
Jul 24 23:26:30.039885 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.039725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-plugins-config-volume\") pod \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\" (UID: \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\") "
Jul 24 23:26:30.039885 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.039743    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvnjw\" (UniqueName: \"kubernetes.io/projected/8cabd655-bbcb-45e7-ad7d-d574e41ab860-kube-api-access-mvnjw\") pod \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\" (UID: \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\") "
Jul 24 23:26:30.039885 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.039773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwqmx\" (UniqueName: \"kubernetes.io/projected/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-kube-api-access-zwqmx\") pod \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\" (UID: \"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d\") "
Jul 24 23:26:30.039885 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.039863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-openshift-service-ca\") pod \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\" (UID: \"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c\") "
Jul 24 23:26:30.040157 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.040105    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cabd655-bbcb-45e7-ad7d-d574e41ab860-plugins-config-volume\") pod \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\" (UID: \"8cabd655-bbcb-45e7-ad7d-d574e41ab860\") "
Jul 24 23:26:30.040208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.040141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d87cf9a2-7062-45e5-b5d5-fee3f0627d8d" (UID: "d87cf9a2-7062-45e5-b5d5-fee3f0627d8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:26:30.040208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.040186    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" (UID: "d82457d9-1ec0-4dff-a6c0-e9d4d114e21c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:26:30.040558 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.040501    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:30.040558 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.040553    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:30.040727 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.040572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cabd655-bbcb-45e7-ad7d-d574e41ab860-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cabd655-bbcb-45e7-ad7d-d574e41ab860" (UID: "8cabd655-bbcb-45e7-ad7d-d574e41ab860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:26:30.041941 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.041920    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cabd655-bbcb-45e7-ad7d-d574e41ab860-kube-api-access-mvnjw" (OuterVolumeSpecName: "kube-api-access-mvnjw") pod "8cabd655-bbcb-45e7-ad7d-d574e41ab860" (UID: "8cabd655-bbcb-45e7-ad7d-d574e41ab860"). InnerVolumeSpecName "kube-api-access-mvnjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:26:30.042343 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.042320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-kube-api-access-smtz5" (OuterVolumeSpecName: "kube-api-access-smtz5") pod "d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" (UID: "d82457d9-1ec0-4dff-a6c0-e9d4d114e21c"). InnerVolumeSpecName "kube-api-access-smtz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:26:30.042402 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.042331    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" (UID: "d82457d9-1ec0-4dff-a6c0-e9d4d114e21c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:26:30.042402 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.042342    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-kube-api-access-zwqmx" (OuterVolumeSpecName: "kube-api-access-zwqmx") pod "d87cf9a2-7062-45e5-b5d5-fee3f0627d8d" (UID: "d87cf9a2-7062-45e5-b5d5-fee3f0627d8d"). InnerVolumeSpecName "kube-api-access-zwqmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:26:30.062544 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.062487    2563 generic.go:356] "Generic (PLEG): container finished" podID="d87cf9a2-7062-45e5-b5d5-fee3f0627d8d" containerID="4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80" exitCode=0
Jul 24 23:26:30.062734 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.062550    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"
Jul 24 23:26:30.062734 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.062553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd" event={"ID":"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d","Type":"ContainerDied","Data":"4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80"}
Jul 24 23:26:30.062734 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.062616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd" event={"ID":"d87cf9a2-7062-45e5-b5d5-fee3f0627d8d","Type":"ContainerDied","Data":"9917ba6cc29b001f2d0b8a83056dcfd3f2afd843051733261cc0976376580471"}
Jul 24 23:26:30.062734 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.062639    2563 scope.go:117] "RemoveContainer" containerID="4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80"
Jul 24 23:26:30.064143 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.064117    2563 generic.go:356] "Generic (PLEG): container finished" podID="d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" containerID="ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a" exitCode=0
Jul 24 23:26:30.064265 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.064168    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"
Jul 24 23:26:30.064265 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.064200    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts" event={"ID":"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c","Type":"ContainerDied","Data":"ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a"}
Jul 24 23:26:30.064265 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.064244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts" event={"ID":"d82457d9-1ec0-4dff-a6c0-e9d4d114e21c","Type":"ContainerDied","Data":"c74697b5844ef46f097314780aa0b3cb4912b19acfc6a6467b73b5b4e3ed88db"}
Jul 24 23:26:30.065582 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.065558    2563 generic.go:356] "Generic (PLEG): container finished" podID="8cabd655-bbcb-45e7-ad7d-d574e41ab860" containerID="bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301" exitCode=0
Jul 24 23:26:30.065692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.065599    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"
Jul 24 23:26:30.065692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.065633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh" event={"ID":"8cabd655-bbcb-45e7-ad7d-d574e41ab860","Type":"ContainerDied","Data":"bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301"}
Jul 24 23:26:30.065790 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.065690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh" event={"ID":"8cabd655-bbcb-45e7-ad7d-d574e41ab860","Type":"ContainerDied","Data":"68cd1e40489f2be792b8431c4391d7d1cca29972289ccc71413cb3360b91cb9c"}
Jul 24 23:26:30.074596 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.074495    2563 scope.go:117] "RemoveContainer" containerID="4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80"
Jul 24 23:26:30.075196 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:26:30.075171    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80\": container with ID starting with 4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80 not found: ID does not exist" containerID="4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80"
Jul 24 23:26:30.075306 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.075210    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80"} err="failed to get container status \"4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80\": rpc error: code = NotFound desc = could not find container \"4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80\": container with ID starting with 4c08b3620438b09d12700ad2a6d1e1081f260c596f5cdc47e9a7d9991b37aa80 not found: ID does not exist"
Jul 24 23:26:30.075306 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.075236    2563 scope.go:117] "RemoveContainer" containerID="ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a"
Jul 24 23:26:30.088138 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.088117    2563 scope.go:117] "RemoveContainer" containerID="ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a"
Jul 24 23:26:30.088417 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:26:30.088397    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a\": container with ID starting with ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a not found: ID does not exist" containerID="ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a"
Jul 24 23:26:30.088463 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.088426    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a"} err="failed to get container status \"ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a\": rpc error: code = NotFound desc = could not find container \"ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a\": container with ID starting with ad7576a3b999b217b27471138976153b5ed6dbdd1b1488e7dc0ba19df2bc2a3a not found: ID does not exist"
Jul 24 23:26:30.088463 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.088442    2563 scope.go:117] "RemoveContainer" containerID="bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301"
Jul 24 23:26:30.093393 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.093365    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"]
Jul 24 23:26:30.095886 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.095861    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4ab9b5-5456bd5874-vdngd"]
Jul 24 23:26:30.097100 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.097084    2563 scope.go:117] "RemoveContainer" containerID="bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301"
Jul 24 23:26:30.097395 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:26:30.097374    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301\": container with ID starting with bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301 not found: ID does not exist" containerID="bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301"
Jul 24 23:26:30.097485 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.097402    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301"} err="failed to get container status \"bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301\": rpc error: code = NotFound desc = could not find container \"bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301\": container with ID starting with bb3d05d0e6f18ed62f39f8b20c4f21f8a9a9962fd26a93a3dbfe79fe3373e301 not found: ID does not exist"
Jul 24 23:26:30.105623 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.105594    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"]
Jul 24 23:26:30.108944 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.108918    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4ab9b5-6fb8ddd964-7g2lh"]
Jul 24 23:26:30.119079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.119049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"]
Jul 24 23:26:30.122112 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.122082    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ab9b5-7b9c8d759c-6wjts"]
Jul 24 23:26:30.141997 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.141957    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cabd655-bbcb-45e7-ad7d-d574e41ab860-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:30.141997 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.141991    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:30.141997 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.142004    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smtz5\" (UniqueName: \"kubernetes.io/projected/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c-kube-api-access-smtz5\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:30.142397 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.142012    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvnjw\" (UniqueName: \"kubernetes.io/projected/8cabd655-bbcb-45e7-ad7d-d574e41ab860-kube-api-access-mvnjw\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:30.142397 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:30.142023    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqmx\" (UniqueName: \"kubernetes.io/projected/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d-kube-api-access-zwqmx\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:31.840863 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:31.840834    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cabd655-bbcb-45e7-ad7d-d574e41ab860" path="/var/lib/kubelet/pods/8cabd655-bbcb-45e7-ad7d-d574e41ab860/volumes"
Jul 24 23:26:31.841257 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:31.841206    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" path="/var/lib/kubelet/pods/d82457d9-1ec0-4dff-a6c0-e9d4d114e21c/volumes"
Jul 24 23:26:31.841507 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:31.841494    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d87cf9a2-7062-45e5-b5d5-fee3f0627d8d" path="/var/lib/kubelet/pods/d87cf9a2-7062-45e5-b5d5-fee3f0627d8d/volumes"
Jul 24 23:26:32.534109 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:32.534075    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:26:33.981379 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:33.981348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:34.361170 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:34.361104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:39.973005 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:39.972970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:40.071834 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.071801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:40.234869 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.234776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:40.457216 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.457184    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:40.577144 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577066    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"]
Jul 24 23:26:40.577402 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577390    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cabd655-bbcb-45e7-ad7d-d574e41ab860" containerName="payload-pre-processing"
Jul 24 23:26:40.577495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577404    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cabd655-bbcb-45e7-ad7d-d574e41ab860" containerName="payload-pre-processing"
Jul 24 23:26:40.577495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577417    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d87cf9a2-7062-45e5-b5d5-fee3f0627d8d" containerName="payload-processing"
Jul 24 23:26:40.577495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577423    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d87cf9a2-7062-45e5-b5d5-fee3f0627d8d" containerName="payload-processing"
Jul 24 23:26:40.577495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577431    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" containerName="maas-api"
Jul 24 23:26:40.577495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577437    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" containerName="maas-api"
Jul 24 23:26:40.577495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577492    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d87cf9a2-7062-45e5-b5d5-fee3f0627d8d" containerName="payload-processing"
Jul 24 23:26:40.577713 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577501    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d82457d9-1ec0-4dff-a6c0-e9d4d114e21c" containerName="maas-api"
Jul 24 23:26:40.577713 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.577507    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cabd655-bbcb-45e7-ad7d-d574e41ab860" containerName="payload-pre-processing"
Jul 24 23:26:40.580471 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.580454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"
Jul 24 23:26:40.583074 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.583058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:26:40.586679 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.586653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"]
Jul 24 23:26:40.633772 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.633741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b29891-8118-4f39-928e-7b3de7d31f7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-569db3-cv6mw\" (UID: \"08b29891-8118-4f39-928e-7b3de7d31f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"
Jul 24 23:26:40.734440 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.734400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b29891-8118-4f39-928e-7b3de7d31f7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-569db3-cv6mw\" (UID: \"08b29891-8118-4f39-928e-7b3de7d31f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"
Jul 24 23:26:40.735031 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.735014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b29891-8118-4f39-928e-7b3de7d31f7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-569db3-cv6mw\" (UID: \"08b29891-8118-4f39-928e-7b3de7d31f7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"
Jul 24 23:26:40.891856 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:40.891818    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"
Jul 24 23:26:41.050544 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:41.045000    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"]
Jul 24 23:26:41.124499 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:41.124468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw" event={"ID":"08b29891-8118-4f39-928e-7b3de7d31f7e","Type":"ContainerStarted","Data":"37e2a2ebb910883f1331a61015bee30dce5604fca58e85bb1be1063cd8b21fd8"}
Jul 24 23:26:42.131042 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:42.131005    2563 generic.go:356] "Generic (PLEG): container finished" podID="08b29891-8118-4f39-928e-7b3de7d31f7e" containerID="1f7ba4bd15b3586d4185b659945be004987179d5a4dd7b62babe38ee53a9b5f5" exitCode=0
Jul 24 23:26:42.131434 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:42.131051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw" event={"ID":"08b29891-8118-4f39-928e-7b3de7d31f7e","Type":"ContainerDied","Data":"1f7ba4bd15b3586d4185b659945be004987179d5a4dd7b62babe38ee53a9b5f5"}
Jul 24 23:26:43.279386 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:43.279360    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"
Jul 24 23:26:43.357733 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:43.357698    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b29891-8118-4f39-928e-7b3de7d31f7e-maas-api-service-ca\") pod \"08b29891-8118-4f39-928e-7b3de7d31f7e\" (UID: \"08b29891-8118-4f39-928e-7b3de7d31f7e\") "
Jul 24 23:26:43.358035 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:43.358011    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b29891-8118-4f39-928e-7b3de7d31f7e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08b29891-8118-4f39-928e-7b3de7d31f7e" (UID: "08b29891-8118-4f39-928e-7b3de7d31f7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:26:43.459189 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:43.459091    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b29891-8118-4f39-928e-7b3de7d31f7e-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:44.142586 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:44.142503    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"
Jul 24 23:26:44.142728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:44.142501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw" event={"ID":"08b29891-8118-4f39-928e-7b3de7d31f7e","Type":"ContainerDied","Data":"37e2a2ebb910883f1331a61015bee30dce5604fca58e85bb1be1063cd8b21fd8"}
Jul 24 23:26:44.142728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:44.142649    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37e2a2ebb910883f1331a61015bee30dce5604fca58e85bb1be1063cd8b21fd8"
Jul 24 23:26:55.967393 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:55.967309    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"]
Jul 24 23:26:55.967850 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:55.967586    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt" podUID="25e76a50-a90f-4251-b4b6-786cdda47423" containerName="maas-api" containerID="cri-o://42e204b02715beae6a1869dcdc5976455f575b83f35f2d93f9bb5a28c817500e" gracePeriod=30
Jul 24 23:26:55.989099 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:55.989053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:56.012837 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.012795    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"]
Jul 24 23:26:56.013128 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.013069    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx" podUID="b6278e82-02ac-415c-a54b-92e45a425cf9" containerName="payload-processing" containerID="cri-o://fd95e357e764041b8e0d8bc067dd6f99374665b5870d268f504a03a27fbca22a" gracePeriod=30
Jul 24 23:26:56.022745 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.022714    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"]
Jul 24 23:26:56.023252 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.022986    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9" podUID="4ca0fefa-47fc-483d-950c-33c262d61a9e" containerName="payload-pre-processing" containerID="cri-o://0b11ab1bef90ee0ae45c7798f78ce7e847b1c0ebe76c81c514a905ee308c0e06" gracePeriod=30
Jul 24 23:26:56.049917 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.049884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:56.104478 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.104446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:26:56.212078 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.212045    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ca0fefa-47fc-483d-950c-33c262d61a9e" containerID="0b11ab1bef90ee0ae45c7798f78ce7e847b1c0ebe76c81c514a905ee308c0e06" exitCode=0
Jul 24 23:26:56.212257 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.212116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9" event={"ID":"4ca0fefa-47fc-483d-950c-33c262d61a9e","Type":"ContainerDied","Data":"0b11ab1bef90ee0ae45c7798f78ce7e847b1c0ebe76c81c514a905ee308c0e06"}
Jul 24 23:26:56.213854 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.213831    2563 generic.go:356] "Generic (PLEG): container finished" podID="25e76a50-a90f-4251-b4b6-786cdda47423" containerID="42e204b02715beae6a1869dcdc5976455f575b83f35f2d93f9bb5a28c817500e" exitCode=0
Jul 24 23:26:56.213995 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.213902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt" event={"ID":"25e76a50-a90f-4251-b4b6-786cdda47423","Type":"ContainerDied","Data":"42e204b02715beae6a1869dcdc5976455f575b83f35f2d93f9bb5a28c817500e"}
Jul 24 23:26:56.215427 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.215401    2563 generic.go:356] "Generic (PLEG): container finished" podID="b6278e82-02ac-415c-a54b-92e45a425cf9" containerID="fd95e357e764041b8e0d8bc067dd6f99374665b5870d268f504a03a27fbca22a" exitCode=0
Jul 24 23:26:56.215568 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.215469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx" event={"ID":"b6278e82-02ac-415c-a54b-92e45a425cf9","Type":"ContainerDied","Data":"fd95e357e764041b8e0d8bc067dd6f99374665b5870d268f504a03a27fbca22a"}
Jul 24 23:26:56.293659 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.293634    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:56.295654 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.295632    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:56.327232 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.327205    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:56.370950 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.370918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6278e82-02ac-415c-a54b-92e45a425cf9-plugins-config-volume\") pod \"b6278e82-02ac-415c-a54b-92e45a425cf9\" (UID: \"b6278e82-02ac-415c-a54b-92e45a425cf9\") "
Jul 24 23:26:56.371106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.370971    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szdcg\" (UniqueName: \"kubernetes.io/projected/25e76a50-a90f-4251-b4b6-786cdda47423-kube-api-access-szdcg\") pod \"25e76a50-a90f-4251-b4b6-786cdda47423\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") "
Jul 24 23:26:56.371106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.370991    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e76a50-a90f-4251-b4b6-786cdda47423-openshift-service-ca\") pod \"25e76a50-a90f-4251-b4b6-786cdda47423\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") "
Jul 24 23:26:56.371106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.371035    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25e76a50-a90f-4251-b4b6-786cdda47423-maas-api-tls\") pod \"25e76a50-a90f-4251-b4b6-786cdda47423\" (UID: \"25e76a50-a90f-4251-b4b6-786cdda47423\") "
Jul 24 23:26:56.371268 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.371109    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvptb\" (UniqueName: \"kubernetes.io/projected/b6278e82-02ac-415c-a54b-92e45a425cf9-kube-api-access-dvptb\") pod \"b6278e82-02ac-415c-a54b-92e45a425cf9\" (UID: \"b6278e82-02ac-415c-a54b-92e45a425cf9\") "
Jul 24 23:26:56.371373 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.371351    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e76a50-a90f-4251-b4b6-786cdda47423-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25e76a50-a90f-4251-b4b6-786cdda47423" (UID: "25e76a50-a90f-4251-b4b6-786cdda47423"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:26:56.371444 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.371385    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6278e82-02ac-415c-a54b-92e45a425cf9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6278e82-02ac-415c-a54b-92e45a425cf9" (UID: "b6278e82-02ac-415c-a54b-92e45a425cf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:26:56.371495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.371467    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6278e82-02ac-415c-a54b-92e45a425cf9-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:56.371495 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.371485    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25e76a50-a90f-4251-b4b6-786cdda47423-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:56.373263 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.373234    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6278e82-02ac-415c-a54b-92e45a425cf9-kube-api-access-dvptb" (OuterVolumeSpecName: "kube-api-access-dvptb") pod "b6278e82-02ac-415c-a54b-92e45a425cf9" (UID: "b6278e82-02ac-415c-a54b-92e45a425cf9"). InnerVolumeSpecName "kube-api-access-dvptb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:26:56.373372 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.373269    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e76a50-a90f-4251-b4b6-786cdda47423-kube-api-access-szdcg" (OuterVolumeSpecName: "kube-api-access-szdcg") pod "25e76a50-a90f-4251-b4b6-786cdda47423" (UID: "25e76a50-a90f-4251-b4b6-786cdda47423"). InnerVolumeSpecName "kube-api-access-szdcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:26:56.373372 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.373270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25e76a50-a90f-4251-b4b6-786cdda47423-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25e76a50-a90f-4251-b4b6-786cdda47423" (UID: "25e76a50-a90f-4251-b4b6-786cdda47423"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:26:56.472628 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.472581    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/4ca0fefa-47fc-483d-950c-33c262d61a9e-kube-api-access-wtt6q\") pod \"4ca0fefa-47fc-483d-950c-33c262d61a9e\" (UID: \"4ca0fefa-47fc-483d-950c-33c262d61a9e\") "
Jul 24 23:26:56.472825 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.472704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca0fefa-47fc-483d-950c-33c262d61a9e-plugins-config-volume\") pod \"4ca0fefa-47fc-483d-950c-33c262d61a9e\" (UID: \"4ca0fefa-47fc-483d-950c-33c262d61a9e\") "
Jul 24 23:26:56.472893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.472837    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szdcg\" (UniqueName: \"kubernetes.io/projected/25e76a50-a90f-4251-b4b6-786cdda47423-kube-api-access-szdcg\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:56.472893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.472848    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25e76a50-a90f-4251-b4b6-786cdda47423-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:56.472893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.472858    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvptb\" (UniqueName: \"kubernetes.io/projected/b6278e82-02ac-415c-a54b-92e45a425cf9-kube-api-access-dvptb\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:56.473137 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.473110    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca0fefa-47fc-483d-950c-33c262d61a9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ca0fefa-47fc-483d-950c-33c262d61a9e" (UID: "4ca0fefa-47fc-483d-950c-33c262d61a9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:26:56.474732 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.474676    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca0fefa-47fc-483d-950c-33c262d61a9e-kube-api-access-wtt6q" (OuterVolumeSpecName: "kube-api-access-wtt6q") pod "4ca0fefa-47fc-483d-950c-33c262d61a9e" (UID: "4ca0fefa-47fc-483d-950c-33c262d61a9e"). InnerVolumeSpecName "kube-api-access-wtt6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:26:56.573274 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.573234    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ca0fefa-47fc-483d-950c-33c262d61a9e-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:56.573274 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:56.573269    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/4ca0fefa-47fc-483d-950c-33c262d61a9e-kube-api-access-wtt6q\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:26:57.221906 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.221876    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"
Jul 24 23:26:57.222330 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.221930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx" event={"ID":"b6278e82-02ac-415c-a54b-92e45a425cf9","Type":"ContainerDied","Data":"5c7fb25494086c614909cd22694eda54943f45b96b42d5de473453c7c0c03537"}
Jul 24 23:26:57.222330 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.221995    2563 scope.go:117] "RemoveContainer" containerID="fd95e357e764041b8e0d8bc067dd6f99374665b5870d268f504a03a27fbca22a"
Jul 24 23:26:57.223433 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.223417    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"
Jul 24 23:26:57.223575 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.223428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9" event={"ID":"4ca0fefa-47fc-483d-950c-33c262d61a9e","Type":"ContainerDied","Data":"81d729b3ffd10280e30017fc6c6f0163441c6d892e3b65be32c7f6fc430e914a"}
Jul 24 23:26:57.224977 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.224952    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt" event={"ID":"25e76a50-a90f-4251-b4b6-786cdda47423","Type":"ContainerDied","Data":"bd3fe3e9f44918ab1d38efc8970e6af142d110326c9c026bc8dffe9acc9d8df0"}
Jul 24 23:26:57.224977 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.224971    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"
Jul 24 23:26:57.231554 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.231423    2563 scope.go:117] "RemoveContainer" containerID="0b11ab1bef90ee0ae45c7798f78ce7e847b1c0ebe76c81c514a905ee308c0e06"
Jul 24 23:26:57.240141 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.240125    2563 scope.go:117] "RemoveContainer" containerID="42e204b02715beae6a1869dcdc5976455f575b83f35f2d93f9bb5a28c817500e"
Jul 24 23:26:57.247989 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.247968    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"]
Jul 24 23:26:57.251203 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.251182    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-569db3-5d846c76d4-khtsx"]
Jul 24 23:26:57.261149 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.261123    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"]
Jul 24 23:26:57.264506 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.264482    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-569db3-69b768764-v5vt9"]
Jul 24 23:26:57.274931 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.274901    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"]
Jul 24 23:26:57.276768 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.276743    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-569db3-788889b5b4-hfhxt"]
Jul 24 23:26:57.841256 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.841213    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25e76a50-a90f-4251-b4b6-786cdda47423" path="/var/lib/kubelet/pods/25e76a50-a90f-4251-b4b6-786cdda47423/volumes"
Jul 24 23:26:57.841720 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.841700    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca0fefa-47fc-483d-950c-33c262d61a9e" path="/var/lib/kubelet/pods/4ca0fefa-47fc-483d-950c-33c262d61a9e/volumes"
Jul 24 23:26:57.842071 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:26:57.842054    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6278e82-02ac-415c-a54b-92e45a425cf9" path="/var/lib/kubelet/pods/b6278e82-02ac-415c-a54b-92e45a425cf9/volumes"
Jul 24 23:27:00.648861 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:00.648823    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"]
Jul 24 23:27:00.651503 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:00.651479    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-569db3-cv6mw"]
Jul 24 23:27:00.651733 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:00.651715    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:00.922693 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:00.922595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:27:01.036464 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:01.036429    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:01.841966 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:01.841933    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b29891-8118-4f39-928e-7b3de7d31f7e" path="/var/lib/kubelet/pods/08b29891-8118-4f39-928e-7b3de7d31f7e/volumes"
Jul 24 23:27:06.644651 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:06.644614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:06.704206 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:06.704172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:06.895444 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:06.895359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:07.132232 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:07.132198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:07.477688 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:07.477656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:07.781380 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:07.781294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:08.315887 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:08.315856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:10.877148 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.877108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:10.972594 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.972563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:10.996696 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.996666    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"]
Jul 24 23:27:10.996982 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.996965    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6278e82-02ac-415c-a54b-92e45a425cf9" containerName="payload-processing"
Jul 24 23:27:10.996982 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.996979    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6278e82-02ac-415c-a54b-92e45a425cf9" containerName="payload-processing"
Jul 24 23:27:10.997079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.996993    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25e76a50-a90f-4251-b4b6-786cdda47423" containerName="maas-api"
Jul 24 23:27:10.997079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.996999    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25e76a50-a90f-4251-b4b6-786cdda47423" containerName="maas-api"
Jul 24 23:27:10.997079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.997009    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b29891-8118-4f39-928e-7b3de7d31f7e" containerName="revoke-keys"
Jul 24 23:27:10.997079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.997014    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b29891-8118-4f39-928e-7b3de7d31f7e" containerName="revoke-keys"
Jul 24 23:27:10.997079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.997023    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca0fefa-47fc-483d-950c-33c262d61a9e" containerName="payload-pre-processing"
Jul 24 23:27:10.997079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.997028    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca0fefa-47fc-483d-950c-33c262d61a9e" containerName="payload-pre-processing"
Jul 24 23:27:10.997079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.997077    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25e76a50-a90f-4251-b4b6-786cdda47423" containerName="maas-api"
Jul 24 23:27:10.997318 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.997084    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b29891-8118-4f39-928e-7b3de7d31f7e" containerName="revoke-keys"
Jul 24 23:27:10.997318 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.997093    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca0fefa-47fc-483d-950c-33c262d61a9e" containerName="payload-pre-processing"
Jul 24 23:27:10.997318 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:10.997099    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6278e82-02ac-415c-a54b-92e45a425cf9" containerName="payload-processing"
Jul 24 23:27:11.001047 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.001026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.003950 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.003929    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-5c850d-serving-cert\""
Jul 24 23:27:11.004064 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.003955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\""
Jul 24 23:27:11.004064 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.003979    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 23:27:11.005164 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.005142    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 23:27:11.008967 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.008943    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"]
Jul 24 23:27:11.090157 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.090129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:11.103334 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.103304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svlnz\" (UniqueName: \"kubernetes.io/projected/144d2566-8c25-419b-930f-c64fa9def932-kube-api-access-svlnz\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.103548 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.103356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/144d2566-8c25-419b-930f-c64fa9def932-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.103548 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.103418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144d2566-8c25-419b-930f-c64fa9def932-maas-api-tls\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.193411 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.193328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:11.204458 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.204426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/144d2566-8c25-419b-930f-c64fa9def932-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.204612 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.204472    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144d2566-8c25-419b-930f-c64fa9def932-maas-api-tls\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.204612 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.204534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svlnz\" (UniqueName: \"kubernetes.io/projected/144d2566-8c25-419b-930f-c64fa9def932-kube-api-access-svlnz\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.205085 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.205067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/144d2566-8c25-419b-930f-c64fa9def932-openshift-service-ca\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.206963 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.206934    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144d2566-8c25-419b-930f-c64fa9def932-maas-api-tls\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.212645 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.212621    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svlnz\" (UniqueName: \"kubernetes.io/projected/144d2566-8c25-419b-930f-c64fa9def932-kube-api-access-svlnz\") pod \"maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.312637 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.312594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:11.447833 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.447809    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"]
Jul 24 23:27:11.449604 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:27:11.449567    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144d2566_8c25_419b_930f_c64fa9def932.slice/crio-7e1e5320c462553bc568c7a262359694b2b8b50ef3419b7917c087810d64e5f7 WatchSource:0}: Error finding container 7e1e5320c462553bc568c7a262359694b2b8b50ef3419b7917c087810d64e5f7: Status 404 returned error can't find the container with id 7e1e5320c462553bc568c7a262359694b2b8b50ef3419b7917c087810d64e5f7
Jul 24 23:27:11.566227 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.566195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:11.658217 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.658184    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"]
Jul 24 23:27:11.661750 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.661726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:11.662928 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.662906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:11.665811 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.665793    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-5c850d-dockercfg-zpthd\""
Jul 24 23:27:11.665902 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.665799    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-5c850d\""
Jul 24 23:27:11.672328 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.672305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"]
Jul 24 23:27:11.701545 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.701453    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"]
Jul 24 23:27:11.704833 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.704816    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:11.709567 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.709540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vp9s\" (UniqueName: \"kubernetes.io/projected/9c886d69-8bb4-4e02-b966-aa28d31d36dc-kube-api-access-7vp9s\") pod \"payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c\" (UID: \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:11.709678 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.709586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c886d69-8bb4-4e02-b966-aa28d31d36dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c\" (UID: \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:11.709678 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.709627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km22t\" (UniqueName: \"kubernetes.io/projected/d19efbd9-82bf-4a27-8f9a-c542633bbac3-kube-api-access-km22t\") pod \"payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j\" (UID: \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:11.709678 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.709655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19efbd9-82bf-4a27-8f9a-c542633bbac3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j\" (UID: \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:11.714304 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.714277    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"]
Jul 24 23:27:11.810254 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.810216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vp9s\" (UniqueName: \"kubernetes.io/projected/9c886d69-8bb4-4e02-b966-aa28d31d36dc-kube-api-access-7vp9s\") pod \"payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c\" (UID: \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:11.810254 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.810258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c886d69-8bb4-4e02-b966-aa28d31d36dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c\" (UID: \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:11.810494 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.810374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km22t\" (UniqueName: \"kubernetes.io/projected/d19efbd9-82bf-4a27-8f9a-c542633bbac3-kube-api-access-km22t\") pod \"payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j\" (UID: \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:11.810494 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.810419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19efbd9-82bf-4a27-8f9a-c542633bbac3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j\" (UID: \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:11.810979 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.810962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c886d69-8bb4-4e02-b966-aa28d31d36dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c\" (UID: \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:11.811154 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.811138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19efbd9-82bf-4a27-8f9a-c542633bbac3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j\" (UID: \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:11.818604 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.818586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km22t\" (UniqueName: \"kubernetes.io/projected/d19efbd9-82bf-4a27-8f9a-c542633bbac3-kube-api-access-km22t\") pod \"payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j\" (UID: \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:11.818700 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.818603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vp9s\" (UniqueName: \"kubernetes.io/projected/9c886d69-8bb4-4e02-b966-aa28d31d36dc-kube-api-access-7vp9s\") pod \"payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c\" (UID: \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:11.974283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:11.974187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:12.016281 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.016241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:12.140301 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.138801    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"]
Jul 24 23:27:12.177632 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.177600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"]
Jul 24 23:27:12.182718 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:27:12.182691    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c886d69_8bb4_4e02_b966_aa28d31d36dc.slice/crio-f16bb83f11f18601552e23d7ae94c639273acaae93dc9f97d366aa4528d723dd WatchSource:0}: Error finding container f16bb83f11f18601552e23d7ae94c639273acaae93dc9f97d366aa4528d723dd: Status 404 returned error can't find the container with id f16bb83f11f18601552e23d7ae94c639273acaae93dc9f97d366aa4528d723dd
Jul 24 23:27:12.301422 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.301383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j" event={"ID":"d19efbd9-82bf-4a27-8f9a-c542633bbac3","Type":"ContainerStarted","Data":"9f643234b81ebcafdca5f44076f5f88ae146e55ce811a681367f28646edeadfb"}
Jul 24 23:27:12.301708 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.301450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j" event={"ID":"d19efbd9-82bf-4a27-8f9a-c542633bbac3","Type":"ContainerStarted","Data":"30249148d430b1829b9da1c26ed5cc3ab98ff258f22504e09df873482b474cdb"}
Jul 24 23:27:12.301784 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.301703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:12.303948 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.303915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs" event={"ID":"144d2566-8c25-419b-930f-c64fa9def932","Type":"ContainerStarted","Data":"5d86b7634e29deab0fef7c7b64b538a3ab7a29b7e139c17fa4a8769c4d5a9b9f"}
Jul 24 23:27:12.304091 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.303976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs" event={"ID":"144d2566-8c25-419b-930f-c64fa9def932","Type":"ContainerStarted","Data":"7e1e5320c462553bc568c7a262359694b2b8b50ef3419b7917c087810d64e5f7"}
Jul 24 23:27:12.304091 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.304008    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:12.305728 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.305703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c" event={"ID":"9c886d69-8bb4-4e02-b966-aa28d31d36dc","Type":"ContainerStarted","Data":"a037817cae8d8678feea24f87ac9171ecd57ebb12170362a478d62f16d3a0702"}
Jul 24 23:27:12.305834 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.305744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c" event={"ID":"9c886d69-8bb4-4e02-b966-aa28d31d36dc","Type":"ContainerStarted","Data":"f16bb83f11f18601552e23d7ae94c639273acaae93dc9f97d366aa4528d723dd"}
Jul 24 23:27:12.305834 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.305766    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:12.317948 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.317906    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j" podStartSLOduration=1.3178925000000001 podStartE2EDuration="1.3178925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:12.31682362 +0000 UTC m=+2448.950102563" watchObservedRunningTime="2026-07-24 23:27:12.3178925 +0000 UTC m=+2448.951171439"
Jul 24 23:27:12.334542 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.334475    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs" podStartSLOduration=1.969797667 podStartE2EDuration="2.334460398s" totalImagesPullingTime="364.662731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:12.333411448 +0000 UTC m=+2448.966690391" watchObservedRunningTime="2026-07-24 23:27:12.334460398 +0000 UTC m=+2448.967739364"
Jul 24 23:27:12.351287 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:12.351231    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c" podStartSLOduration=1.351217602 podStartE2EDuration="1.351217602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:12.349597227 +0000 UTC m=+2448.982876167" watchObservedRunningTime="2026-07-24 23:27:12.351217602 +0000 UTC m=+2448.984496520"
Jul 24 23:27:15.372822 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:15.372791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:15.769980 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:15.769864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:16.023247 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:16.023164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:16.204672 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:16.204644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:18.319197 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:18.319169    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:27:18.957296 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:18.957267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:19.018854 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.018820    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"]
Jul 24 23:27:19.023165 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.023139    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.026071 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.026034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d08a26-serving-cert\""
Jul 24 23:27:19.033621 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.033584    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"]
Jul 24 23:27:19.034160 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.034132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:19.042142 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.042115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:27:19.076534 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.076489    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2fb6f1-350c-438b-892e-173ae07d2965-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.076732 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.076566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rzvj\" (UniqueName: \"kubernetes.io/projected/9f2fb6f1-350c-438b-892e-173ae07d2965-kube-api-access-4rzvj\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.076732 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.076607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.109218 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.109182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:19.178138 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.178095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2fb6f1-350c-438b-892e-173ae07d2965-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.178343 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.178169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rzvj\" (UniqueName: \"kubernetes.io/projected/9f2fb6f1-350c-438b-892e-173ae07d2965-kube-api-access-4rzvj\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.178343 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.178200    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.179776 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:27:19.178801    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-serving-cert: secret "maas-api-e2e-shared-b-d08a26-serving-cert" not found
Jul 24 23:27:19.179776 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:27:19.178900    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls podName:9f2fb6f1-350c-438b-892e-173ae07d2965 nodeName:}" failed. No retries permitted until 2026-07-24 23:27:19.678866783 +0000 UTC m=+2456.312145702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls") pod "maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9" (UID: "9f2fb6f1-350c-438b-892e-173ae07d2965") : secret "maas-api-e2e-shared-b-d08a26-serving-cert" not found
Jul 24 23:27:19.179776 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.179359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2fb6f1-350c-438b-892e-173ae07d2965-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.188726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.188697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rzvj\" (UniqueName: \"kubernetes.io/projected/9f2fb6f1-350c-438b-892e-173ae07d2965-kube-api-access-4rzvj\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.261816 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.261690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:19.630966 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.630923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:19.682738 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.682693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.685299 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.685262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls\") pod \"maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.712288 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.712250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:19.716101 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.716073    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"]
Jul 24 23:27:19.719671 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.719654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:19.722416 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.722396    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-d08a26\""
Jul 24 23:27:19.732172 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.732147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"]
Jul 24 23:27:19.758596 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.758555    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"]
Jul 24 23:27:19.761874 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.761852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:19.764707 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.764684    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-d08a26-dockercfg-hfjr5\""
Jul 24 23:27:19.771372 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.771343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"]
Jul 24 23:27:19.783649 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.783620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmll\" (UniqueName: \"kubernetes.io/projected/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-kube-api-access-stmll\") pod \"payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp\" (UID: \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:19.783809 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.783676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp\" (UID: \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:19.884697 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.884603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wzr\" (UniqueName: \"kubernetes.io/projected/1d5ac741-7edc-4b79-92b5-ed7b300de390-kube-api-access-v7wzr\") pod \"payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp\" (UID: \"1d5ac741-7edc-4b79-92b5-ed7b300de390\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:19.884697 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.884691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp\" (UID: \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:19.884938 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.884767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmll\" (UniqueName: \"kubernetes.io/projected/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-kube-api-access-stmll\") pod \"payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp\" (UID: \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:19.884938 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.884809    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5ac741-7edc-4b79-92b5-ed7b300de390-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp\" (UID: \"1d5ac741-7edc-4b79-92b5-ed7b300de390\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:19.885389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.885366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp\" (UID: \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:19.892919 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.892884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmll\" (UniqueName: \"kubernetes.io/projected/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-kube-api-access-stmll\") pod \"payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp\" (UID: \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:19.936591 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.936549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:19.986005 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.985958    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5ac741-7edc-4b79-92b5-ed7b300de390-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp\" (UID: \"1d5ac741-7edc-4b79-92b5-ed7b300de390\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:19.986129 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.986038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wzr\" (UniqueName: \"kubernetes.io/projected/1d5ac741-7edc-4b79-92b5-ed7b300de390-kube-api-access-v7wzr\") pod \"payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp\" (UID: \"1d5ac741-7edc-4b79-92b5-ed7b300de390\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:19.986903 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.986876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5ac741-7edc-4b79-92b5-ed7b300de390-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp\" (UID: \"1d5ac741-7edc-4b79-92b5-ed7b300de390\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:19.995819 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:19.995787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wzr\" (UniqueName: \"kubernetes.io/projected/1d5ac741-7edc-4b79-92b5-ed7b300de390-kube-api-access-v7wzr\") pod \"payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp\" (UID: \"1d5ac741-7edc-4b79-92b5-ed7b300de390\") " pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:20.029994 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.029908    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:20.073413 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.073378    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:20.082117 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.078975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"]
Jul 24 23:27:20.087071 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:27:20.086933    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2fb6f1_350c_438b_892e_173ae07d2965.slice/crio-0f68942fedf0ee81443817fd89a45ed915f50b2484c3ff09c1abfcb65c543cbc WatchSource:0}: Error finding container 0f68942fedf0ee81443817fd89a45ed915f50b2484c3ff09c1abfcb65c543cbc: Status 404 returned error can't find the container with id 0f68942fedf0ee81443817fd89a45ed915f50b2484c3ff09c1abfcb65c543cbc
Jul 24 23:27:20.176604 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.176579    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"]
Jul 24 23:27:20.178159 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:27:20.178120    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb060cdea_c5cb_4bf3_b91e_edd36b9673c7.slice/crio-88d2dc358145e631b9c717671ae11241486c25e7e419366093c2aa568e983edd WatchSource:0}: Error finding container 88d2dc358145e631b9c717671ae11241486c25e7e419366093c2aa568e983edd: Status 404 returned error can't find the container with id 88d2dc358145e631b9c717671ae11241486c25e7e419366093c2aa568e983edd
Jul 24 23:27:20.233835 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.233808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"]
Jul 24 23:27:20.236757 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:27:20.236725    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5ac741_7edc_4b79_92b5_ed7b300de390.slice/crio-edef63e6b6d0eb1a7df53e9d16f3497f442b80df432a8f09424310f57c562540 WatchSource:0}: Error finding container edef63e6b6d0eb1a7df53e9d16f3497f442b80df432a8f09424310f57c562540: Status 404 returned error can't find the container with id edef63e6b6d0eb1a7df53e9d16f3497f442b80df432a8f09424310f57c562540
Jul 24 23:27:20.351491 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.351453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp" event={"ID":"b060cdea-c5cb-4bf3-b91e-edd36b9673c7","Type":"ContainerStarted","Data":"c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3"}
Jul 24 23:27:20.351719 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.351513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp" event={"ID":"b060cdea-c5cb-4bf3-b91e-edd36b9673c7","Type":"ContainerStarted","Data":"88d2dc358145e631b9c717671ae11241486c25e7e419366093c2aa568e983edd"}
Jul 24 23:27:20.351719 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.351596    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:20.353041 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.353015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9" event={"ID":"9f2fb6f1-350c-438b-892e-173ae07d2965","Type":"ContainerStarted","Data":"0f68942fedf0ee81443817fd89a45ed915f50b2484c3ff09c1abfcb65c543cbc"}
Jul 24 23:27:20.354633 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.354609    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp" event={"ID":"1d5ac741-7edc-4b79-92b5-ed7b300de390","Type":"ContainerStarted","Data":"b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46"}
Jul 24 23:27:20.354757 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.354649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp" event={"ID":"1d5ac741-7edc-4b79-92b5-ed7b300de390","Type":"ContainerStarted","Data":"edef63e6b6d0eb1a7df53e9d16f3497f442b80df432a8f09424310f57c562540"}
Jul 24 23:27:20.354757 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.354669    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:20.368177 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.368123    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp" podStartSLOduration=1.36810648 podStartE2EDuration="1.36810648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:20.366626274 +0000 UTC m=+2456.999905214" watchObservedRunningTime="2026-07-24 23:27:20.36810648 +0000 UTC m=+2457.001385399"
Jul 24 23:27:20.383468 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:20.383411    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp" podStartSLOduration=1.383394043 podStartE2EDuration="1.383394043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:20.381651375 +0000 UTC m=+2457.014930316" watchObservedRunningTime="2026-07-24 23:27:20.383394043 +0000 UTC m=+2457.016672982"
Jul 24 23:27:21.362654 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:21.362616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9" event={"ID":"9f2fb6f1-350c-438b-892e-173ae07d2965","Type":"ContainerStarted","Data":"5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987"}
Jul 24 23:27:21.363089 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:21.362704    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:21.381101 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:21.381047    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9" podStartSLOduration=2.9791112650000002 podStartE2EDuration="3.381027454s" totalImagesPullingTime="401.916189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:21.378974351 +0000 UTC m=+2458.012253296" watchObservedRunningTime="2026-07-24 23:27:21.381027454 +0000 UTC m=+2458.014306434"
Jul 24 23:27:23.314735 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:23.314701    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:27:23.315192 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:23.314889    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:27:23.472016 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:23.471980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:24.157228 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:24.157195    2563 scope.go:117] "RemoveContainer" containerID="5bf586a27bbb7174f79666b5a5c00d4fcf2d8afb429651b5a4a043bb275cef46"
Jul 24 23:27:27.373169 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:27.373137    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:27:31.365087 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:31.365058    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:27:31.365627 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:31.365609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:27:33.532616 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:33.532578    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:27:36.478661 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:36.478626    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:42.728539 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:42.728475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:27:56.176864 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:56.176829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:27:59.632281 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:27:59.632249    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:28:14.631704 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:28:14.631669    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:28:14.644337 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:28:14.644309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:28:14.696662 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:28:14.696631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:28:34.627277 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:28:34.627232    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:28:50.021610 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:28:50.021570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:29:08.821331 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:29:08.821291    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:29:28.230017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:29:28.229975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:29:46.732262 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:29:46.732178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:30:00.135879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.135836    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5"]
Jul 24 23:30:00.139454 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.139433    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"]
Jul 24 23:30:00.139604 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.139549    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5"
Jul 24 23:30:00.142045 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.142016    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:30:00.142647 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.142624    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"]
Jul 24 23:30:00.142744 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.142733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"
Jul 24 23:30:00.145738 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.145719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"
Jul 24 23:30:00.150059 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.150040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5"
Jul 24 23:30:00.154485 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.154462    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"]
Jul 24 23:30:00.159556 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.158565    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"]
Jul 24 23:30:00.161134 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.161113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5"]
Jul 24 23:30:00.162245 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.162222    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"
Jul 24 23:30:00.196998 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.196976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"
Jul 24 23:30:00.322801 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.322763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5"]
Jul 24 23:30:00.324432 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:30:00.324400    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f494848_e793_4b97_9993_f1417d2a2066.slice/crio-914d71803c457acd1174d72b90dd842c1ca05d110dc372f496f89b7a3a6b6564 WatchSource:0}: Error finding container 914d71803c457acd1174d72b90dd842c1ca05d110dc372f496f89b7a3a6b6564: Status 404 returned error can't find the container with id 914d71803c457acd1174d72b90dd842c1ca05d110dc372f496f89b7a3a6b6564
Jul 24 23:30:00.371820 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.371795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"]
Jul 24 23:30:00.373417 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:30:00.373389    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaa525f_cbb1_47db_81fd_f06e1bd067b0.slice/crio-e4ef971ebf34a4ad16e2b0289d48d00610d75ee20e1ddb133aac1c5e8fd6d558 WatchSource:0}: Error finding container e4ef971ebf34a4ad16e2b0289d48d00610d75ee20e1ddb133aac1c5e8fd6d558: Status 404 returned error can't find the container with id e4ef971ebf34a4ad16e2b0289d48d00610d75ee20e1ddb133aac1c5e8fd6d558
Jul 24 23:30:00.546088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:00.546064    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"]
Jul 24 23:30:00.547884 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:30:00.547859    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908a0de9_a26a_4080_8d17_78608a9412ea.slice/crio-f6148e7efcfc7f3c2ebf01799b76fec83980fa5e92eaa4abe65e6d7059b66ec3 WatchSource:0}: Error finding container f6148e7efcfc7f3c2ebf01799b76fec83980fa5e92eaa4abe65e6d7059b66ec3: Status 404 returned error can't find the container with id f6148e7efcfc7f3c2ebf01799b76fec83980fa5e92eaa4abe65e6d7059b66ec3
Jul 24 23:30:01.144041 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.144007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w" event={"ID":"3aaa525f-cbb1-47db-81fd-f06e1bd067b0","Type":"ContainerStarted","Data":"4cfffe3d8d64d5edcd7695217d0f96385a75487066e4aa55bbcf437c6082986b"}
Jul 24 23:30:01.144464 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.144064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w" event={"ID":"3aaa525f-cbb1-47db-81fd-f06e1bd067b0","Type":"ContainerStarted","Data":"e4ef971ebf34a4ad16e2b0289d48d00610d75ee20e1ddb133aac1c5e8fd6d558"}
Jul 24 23:30:01.145580 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.145552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5" event={"ID":"6f494848-e793-4b97-9993-f1417d2a2066","Type":"ContainerStarted","Data":"dc399b958d37a476415dacf5ba33b38df64e0526beca04e55073f844a917aa3b"}
Jul 24 23:30:01.145693 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.145601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5" event={"ID":"6f494848-e793-4b97-9993-f1417d2a2066","Type":"ContainerStarted","Data":"914d71803c457acd1174d72b90dd842c1ca05d110dc372f496f89b7a3a6b6564"}
Jul 24 23:30:01.146903 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.146882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56" event={"ID":"908a0de9-a26a-4080-8d17-78608a9412ea","Type":"ContainerStarted","Data":"ab28fb3d54ee6803f17e0f9041dae22be547ddffcd34e71e443ae796d608385d"}
Jul 24 23:30:01.146994 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.146919    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56" event={"ID":"908a0de9-a26a-4080-8d17-78608a9412ea","Type":"ContainerStarted","Data":"f6148e7efcfc7f3c2ebf01799b76fec83980fa5e92eaa4abe65e6d7059b66ec3"}
Jul 24 23:30:01.158308 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.158269    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w" podStartSLOduration=1.158256303 podStartE2EDuration="1.158256303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:01.157365242 +0000 UTC m=+2617.790644182" watchObservedRunningTime="2026-07-24 23:30:01.158256303 +0000 UTC m=+2617.791535290"
Jul 24 23:30:01.172399 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.172346    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56" podStartSLOduration=1.172328447 podStartE2EDuration="1.172328447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:01.170465952 +0000 UTC m=+2617.803744895" watchObservedRunningTime="2026-07-24 23:30:01.172328447 +0000 UTC m=+2617.805607366"
Jul 24 23:30:01.186839 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:01.186789    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5" podStartSLOduration=1.186770896 podStartE2EDuration="1.186770896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:01.185895609 +0000 UTC m=+2617.819174550" watchObservedRunningTime="2026-07-24 23:30:01.186770896 +0000 UTC m=+2617.820049877"
Jul 24 23:30:02.153400 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:02.153365    2563 generic.go:356] "Generic (PLEG): container finished" podID="3aaa525f-cbb1-47db-81fd-f06e1bd067b0" containerID="4cfffe3d8d64d5edcd7695217d0f96385a75487066e4aa55bbcf437c6082986b" exitCode=0
Jul 24 23:30:02.153861 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:02.153456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w" event={"ID":"3aaa525f-cbb1-47db-81fd-f06e1bd067b0","Type":"ContainerDied","Data":"4cfffe3d8d64d5edcd7695217d0f96385a75487066e4aa55bbcf437c6082986b"}
Jul 24 23:30:02.155022 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:02.154997    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f494848-e793-4b97-9993-f1417d2a2066" containerID="dc399b958d37a476415dacf5ba33b38df64e0526beca04e55073f844a917aa3b" exitCode=0
Jul 24 23:30:02.155135 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:02.155082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5" event={"ID":"6f494848-e793-4b97-9993-f1417d2a2066","Type":"ContainerDied","Data":"dc399b958d37a476415dacf5ba33b38df64e0526beca04e55073f844a917aa3b"}
Jul 24 23:30:02.156558 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:02.156538    2563 generic.go:356] "Generic (PLEG): container finished" podID="908a0de9-a26a-4080-8d17-78608a9412ea" containerID="ab28fb3d54ee6803f17e0f9041dae22be547ddffcd34e71e443ae796d608385d" exitCode=0
Jul 24 23:30:02.156656 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:02.156562    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56" event={"ID":"908a0de9-a26a-4080-8d17-78608a9412ea","Type":"ContainerDied","Data":"ab28fb3d54ee6803f17e0f9041dae22be547ddffcd34e71e443ae796d608385d"}
Jul 24 23:30:03.307850 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:03.307817    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5"
Jul 24 23:30:03.336062 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:03.336037    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"
Jul 24 23:30:03.343090 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:03.343062    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"
Jul 24 23:30:03.605072 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:03.605040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:30:04.167498 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.167468    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"
Jul 24 23:30:04.167498 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.167484    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56" event={"ID":"908a0de9-a26a-4080-8d17-78608a9412ea","Type":"ContainerDied","Data":"f6148e7efcfc7f3c2ebf01799b76fec83980fa5e92eaa4abe65e6d7059b66ec3"}
Jul 24 23:30:04.167764 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.167558    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6148e7efcfc7f3c2ebf01799b76fec83980fa5e92eaa4abe65e6d7059b66ec3"
Jul 24 23:30:04.168843 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.168824    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"
Jul 24 23:30:04.168972 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.168867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w" event={"ID":"3aaa525f-cbb1-47db-81fd-f06e1bd067b0","Type":"ContainerDied","Data":"e4ef971ebf34a4ad16e2b0289d48d00610d75ee20e1ddb133aac1c5e8fd6d558"}
Jul 24 23:30:04.168972 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.168918    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4ef971ebf34a4ad16e2b0289d48d00610d75ee20e1ddb133aac1c5e8fd6d558"
Jul 24 23:30:04.170354 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.170328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5" event={"ID":"6f494848-e793-4b97-9993-f1417d2a2066","Type":"ContainerDied","Data":"914d71803c457acd1174d72b90dd842c1ca05d110dc372f496f89b7a3a6b6564"}
Jul 24 23:30:04.170354 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.170349    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748930-9dgz5"
Jul 24 23:30:04.170513 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.170374    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914d71803c457acd1174d72b90dd842c1ca05d110dc372f496f89b7a3a6b6564"
Jul 24 23:30:04.589164 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:04.589074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:30:05.029768 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:05.029732    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:30:15.081635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:15.081593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:30:15.432049 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:15.432006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:30:25.028154 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:25.028110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:30:37.282544 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:37.282484    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:30:40.930841 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:40.930799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:30:56.033845 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:30:56.033799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:31:15.827765 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:15.827672    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:31:21.415599 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:21.415529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:31:22.297590 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:22.297553    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:31:23.946779 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:23.946751    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:31:23.950418 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:23.950396    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:31:24.985618 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:24.985577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:31:25.085025 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:25.084988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:31:31.584113 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:31.584086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:31:34.833484 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:34.833438    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:31:50.733599 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:31:50.733559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:32:11.486613 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:32:11.486576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:32:15.349455 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:32:15.349418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:32:17.609870 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:32:17.609824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:32:24.232970 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:32:24.232934    2563 scope.go:117] "RemoveContainer" containerID="a81b979886beb6241c3c7fd365946327b366cce22bd80cd6cba651a58158919b"
Jul 24 23:32:29.232001 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:32:29.231963    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:32:44.623346 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:32:44.623300    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:32:49.586377 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:32:49.586267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:32:50.478941 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:32:50.478900    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:00.028559 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:00.028507    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:33:10.186699 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:10.186671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:10.485625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:10.485532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:10.895904 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:10.895866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:13.024939 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.024908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:13.171732 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.171697    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"]
Jul 24 23:33:13.172212 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172195    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908a0de9-a26a-4080-8d17-78608a9412ea" containerName="cleanup"
Jul 24 23:33:13.172212 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172215    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="908a0de9-a26a-4080-8d17-78608a9412ea" containerName="cleanup"
Jul 24 23:33:13.172395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172227    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f494848-e793-4b97-9993-f1417d2a2066" containerName="cleanup"
Jul 24 23:33:13.172395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172235    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f494848-e793-4b97-9993-f1417d2a2066" containerName="cleanup"
Jul 24 23:33:13.172395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172253    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aaa525f-cbb1-47db-81fd-f06e1bd067b0" containerName="cleanup"
Jul 24 23:33:13.172395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172260    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aaa525f-cbb1-47db-81fd-f06e1bd067b0" containerName="cleanup"
Jul 24 23:33:13.172395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172374    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f494848-e793-4b97-9993-f1417d2a2066" containerName="cleanup"
Jul 24 23:33:13.172395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172385    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aaa525f-cbb1-47db-81fd-f06e1bd067b0" containerName="cleanup"
Jul 24 23:33:13.172395 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.172392    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="908a0de9-a26a-4080-8d17-78608a9412ea" containerName="cleanup"
Jul 24 23:33:13.175513 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.175490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.178009 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.177975    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:13.178296 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.178276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-eb4c8d-serving-cert\""
Jul 24 23:33:13.184870 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.184513    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"]
Jul 24 23:33:13.210325 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.210292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90031a1f-a8a1-4453-a921-f5ca99b9d017-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.210563 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.210393    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljgl\" (UniqueName: \"kubernetes.io/projected/90031a1f-a8a1-4453-a921-f5ca99b9d017-kube-api-access-fljgl\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.210563 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.210430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.248843 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.248809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:13.311431 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.311341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljgl\" (UniqueName: \"kubernetes.io/projected/90031a1f-a8a1-4453-a921-f5ca99b9d017-kube-api-access-fljgl\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.311431 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.311387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.311431 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.311428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90031a1f-a8a1-4453-a921-f5ca99b9d017-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.311729 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:33:13.311579    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-serving-cert: secret "maas-api-e2e-ipp-a-eb4c8d-serving-cert" not found
Jul 24 23:33:13.311729 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:33:13.311675    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls podName:90031a1f-a8a1-4453-a921-f5ca99b9d017 nodeName:}" failed. No retries permitted until 2026-07-24 23:33:13.811642293 +0000 UTC m=+2810.444921227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls") pod "maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs" (UID: "90031a1f-a8a1-4453-a921-f5ca99b9d017") : secret "maas-api-e2e-ipp-a-eb4c8d-serving-cert" not found
Jul 24 23:33:13.312046 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.312030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90031a1f-a8a1-4453-a921-f5ca99b9d017-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.324636 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.324611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljgl\" (UniqueName: \"kubernetes.io/projected/90031a1f-a8a1-4453-a921-f5ca99b9d017-kube-api-access-fljgl\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.620792 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.620752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:13.807820 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.807786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:13.815909 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.815881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.818609 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.818584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:13.901145 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.901028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:13.905835 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.905806    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"]
Jul 24 23:33:13.909578 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.909562    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:13.917478 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.917450    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eb4c8d\""
Jul 24 23:33:13.917654 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.917587    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eb4c8d-dockercfg-4v86b\""
Jul 24 23:33:13.921639 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.921614    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"]
Jul 24 23:33:13.934969 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.934937    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"]
Jul 24 23:33:13.938441 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.938423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:13.947392 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:13.947363    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"]
Jul 24 23:33:14.018123 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.018046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9khr\" (UniqueName: \"kubernetes.io/projected/42727336-91de-48e1-ae90-7e867acddb13-kube-api-access-x9khr\") pod \"payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw\" (UID: \"42727336-91de-48e1-ae90-7e867acddb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:14.018383 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.018152    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42727336-91de-48e1-ae90-7e867acddb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw\" (UID: \"42727336-91de-48e1-ae90-7e867acddb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:14.089179 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.089137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:14.119421 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.119386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ms66\" (UniqueName: \"kubernetes.io/projected/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-kube-api-access-5ms66\") pod \"payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6\" (UID: \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:14.119611 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.119439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9khr\" (UniqueName: \"kubernetes.io/projected/42727336-91de-48e1-ae90-7e867acddb13-kube-api-access-x9khr\") pod \"payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw\" (UID: \"42727336-91de-48e1-ae90-7e867acddb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:14.119611 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.119510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6\" (UID: \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:14.119611 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.119572    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42727336-91de-48e1-ae90-7e867acddb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw\" (UID: \"42727336-91de-48e1-ae90-7e867acddb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:14.120293 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.120267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42727336-91de-48e1-ae90-7e867acddb13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw\" (UID: \"42727336-91de-48e1-ae90-7e867acddb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:14.127549 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.127490    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9khr\" (UniqueName: \"kubernetes.io/projected/42727336-91de-48e1-ae90-7e867acddb13-kube-api-access-x9khr\") pod \"payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw\" (UID: \"42727336-91de-48e1-ae90-7e867acddb13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:14.220066 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.220025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6\" (UID: \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:14.220264 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.220129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ms66\" (UniqueName: \"kubernetes.io/projected/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-kube-api-access-5ms66\") pod \"payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6\" (UID: \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:14.220843 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.220821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6\" (UID: \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:14.226760 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.226733    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:14.229398 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.229376    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"]
Jul 24 23:33:14.230839 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.230817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ms66\" (UniqueName: \"kubernetes.io/projected/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-kube-api-access-5ms66\") pod \"payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6\" (UID: \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:14.231333 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:33:14.231299    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90031a1f_a8a1_4453_a921_f5ca99b9d017.slice/crio-4a8e8ab956a447490701b083ec30d956ef9e57f10977bb9953573aea440c5c1b WatchSource:0}: Error finding container 4a8e8ab956a447490701b083ec30d956ef9e57f10977bb9953573aea440c5c1b: Status 404 returned error can't find the container with id 4a8e8ab956a447490701b083ec30d956ef9e57f10977bb9953573aea440c5c1b
Jul 24 23:33:14.233029 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.233013    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 23:33:14.249756 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.249727    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:14.375058 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.375033    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"]
Jul 24 23:33:14.378534 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:33:14.377656    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42727336_91de_48e1_ae90_7e867acddb13.slice/crio-c3d11ee3ef52bf1b1f591add00d574b6bc37a09102d93f347afbf64fe50eebf8 WatchSource:0}: Error finding container c3d11ee3ef52bf1b1f591add00d574b6bc37a09102d93f347afbf64fe50eebf8: Status 404 returned error can't find the container with id c3d11ee3ef52bf1b1f591add00d574b6bc37a09102d93f347afbf64fe50eebf8
Jul 24 23:33:14.402913 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:14.402885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"]
Jul 24 23:33:14.404756 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:33:14.404704    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d71fc1c_a3d8_432f_a474_0894b4a9a06f.slice/crio-e13c9f2c7d28bb0158a64af9415c48f6c4295580745caa9499f092a73ff175e0 WatchSource:0}: Error finding container e13c9f2c7d28bb0158a64af9415c48f6c4295580745caa9499f092a73ff175e0: Status 404 returned error can't find the container with id e13c9f2c7d28bb0158a64af9415c48f6c4295580745caa9499f092a73ff175e0
Jul 24 23:33:15.152436 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.152396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw" event={"ID":"42727336-91de-48e1-ae90-7e867acddb13","Type":"ContainerStarted","Data":"cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197"}
Jul 24 23:33:15.152948 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.152470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw" event={"ID":"42727336-91de-48e1-ae90-7e867acddb13","Type":"ContainerStarted","Data":"c3d11ee3ef52bf1b1f591add00d574b6bc37a09102d93f347afbf64fe50eebf8"}
Jul 24 23:33:15.152948 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.152638    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:15.154642 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.154610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6" event={"ID":"1d71fc1c-a3d8-432f-a474-0894b4a9a06f","Type":"ContainerStarted","Data":"2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311"}
Jul 24 23:33:15.154800 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.154664    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6" event={"ID":"1d71fc1c-a3d8-432f-a474-0894b4a9a06f","Type":"ContainerStarted","Data":"e13c9f2c7d28bb0158a64af9415c48f6c4295580745caa9499f092a73ff175e0"}
Jul 24 23:33:15.154800 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.154740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:15.156365 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.156342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs" event={"ID":"90031a1f-a8a1-4453-a921-f5ca99b9d017","Type":"ContainerStarted","Data":"719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225"}
Jul 24 23:33:15.156500 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.156391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs" event={"ID":"90031a1f-a8a1-4453-a921-f5ca99b9d017","Type":"ContainerStarted","Data":"4a8e8ab956a447490701b083ec30d956ef9e57f10977bb9953573aea440c5c1b"}
Jul 24 23:33:15.156500 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.156423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:15.170342 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.170291    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw" podStartSLOduration=2.170273121 podStartE2EDuration="2.170273121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:33:15.168000745 +0000 UTC m=+2811.801279685" watchObservedRunningTime="2026-07-24 23:33:15.170273121 +0000 UTC m=+2811.803552059"
Jul 24 23:33:15.183068 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.183015    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6" podStartSLOduration=2.182998335 podStartE2EDuration="2.182998335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:33:15.182008007 +0000 UTC m=+2811.815286947" watchObservedRunningTime="2026-07-24 23:33:15.182998335 +0000 UTC m=+2811.816277252"
Jul 24 23:33:15.201015 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:15.200964    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs" podStartSLOduration=1.6317367699999998 podStartE2EDuration="2.200948584s" totalImagesPullingTime="569.211814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:33:15.20003252 +0000 UTC m=+2811.833311473" watchObservedRunningTime="2026-07-24 23:33:15.200948584 +0000 UTC m=+2811.834227523"
Jul 24 23:33:17.473701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:17.473665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:18.047720 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:18.047689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:18.147711 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:18.147672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:18.533985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:18.533946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:33:21.167530 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:21.167493    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:33:24.254079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:24.254039    2563 scope.go:117] "RemoveContainer" containerID="1f7ba4bd15b3586d4185b659945be004987179d5a4dd7b62babe38ee53a9b5f5"
Jul 24 23:33:26.163646 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:26.163615    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:33:26.164112 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:26.163685    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:33:28.416861 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:28.416829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:28.703828 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:28.703747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:28.936945 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:28.936905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:49.680879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.680846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:49.753090 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.753059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:49.786410 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.786377    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"]
Jul 24 23:33:49.790107 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.790077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:49.792839 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.792818    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-064932-serving-cert\""
Jul 24 23:33:49.800666 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.800642    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"]
Jul 24 23:33:49.887940 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.887911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:49.950993 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.950903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:49.951149 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.950998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvdsz\" (UniqueName: \"kubernetes.io/projected/254fac1f-8d2a-48b4-acfd-96a41c354564-kube-api-access-cvdsz\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:49.951149 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:49.951043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fac1f-8d2a-48b4-acfd-96a41c354564-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.051905 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.051866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.052089 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.051916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvdsz\" (UniqueName: \"kubernetes.io/projected/254fac1f-8d2a-48b4-acfd-96a41c354564-kube-api-access-cvdsz\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.052089 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.051949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fac1f-8d2a-48b4-acfd-96a41c354564-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.052089 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:33:50.052051    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-serving-cert: secret "maas-api-e2e-ipp-b-064932-serving-cert" not found
Jul 24 23:33:50.052215 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:33:50.052139    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls podName:254fac1f-8d2a-48b4-acfd-96a41c354564 nodeName:}" failed. No retries permitted until 2026-07-24 23:33:50.552117044 +0000 UTC m=+2847.185395966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls") pod "maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf" (UID: "254fac1f-8d2a-48b4-acfd-96a41c354564") : secret "maas-api-e2e-ipp-b-064932-serving-cert" not found
Jul 24 23:33:50.052625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.052603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fac1f-8d2a-48b4-acfd-96a41c354564-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.063301 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.063275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvdsz\" (UniqueName: \"kubernetes.io/projected/254fac1f-8d2a-48b4-acfd-96a41c354564-kube-api-access-cvdsz\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.331712 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.331627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:50.456905 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.456871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:50.555032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.555000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:50.557210 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.557189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.559836 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.559811    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls\") pod \"maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.560740 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.560721    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"]
Jul 24 23:33:50.564365 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.564345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:50.567233 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.567212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-064932-dockercfg-p6sl4\""
Jul 24 23:33:50.567389 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.567233    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-064932\""
Jul 24 23:33:50.575055 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.575038    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"]
Jul 24 23:33:50.658386 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.658341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp79j\" (UniqueName: \"kubernetes.io/projected/9303319e-47eb-404e-af84-5d0a2ac92a71-kube-api-access-bp79j\") pod \"payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd\" (UID: \"9303319e-47eb-404e-af84-5d0a2ac92a71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:50.658591 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.658478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9303319e-47eb-404e-af84-5d0a2ac92a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd\" (UID: \"9303319e-47eb-404e-af84-5d0a2ac92a71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:50.701707 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.701674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:50.759004 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.758975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp79j\" (UniqueName: \"kubernetes.io/projected/9303319e-47eb-404e-af84-5d0a2ac92a71-kube-api-access-bp79j\") pod \"payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd\" (UID: \"9303319e-47eb-404e-af84-5d0a2ac92a71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:50.759127 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.759046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9303319e-47eb-404e-af84-5d0a2ac92a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd\" (UID: \"9303319e-47eb-404e-af84-5d0a2ac92a71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:50.760177 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.760125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9303319e-47eb-404e-af84-5d0a2ac92a71-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd\" (UID: \"9303319e-47eb-404e-af84-5d0a2ac92a71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:50.769018 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.768964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp79j\" (UniqueName: \"kubernetes.io/projected/9303319e-47eb-404e-af84-5d0a2ac92a71-kube-api-access-bp79j\") pod \"payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd\" (UID: \"9303319e-47eb-404e-af84-5d0a2ac92a71\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:50.836640 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.836615    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"]
Jul 24 23:33:50.838849 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:33:50.838821    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254fac1f_8d2a_48b4_acfd_96a41c354564.slice/crio-060f4a7eebc38180f6438d40611c02ddf1e089fd2397eed0740b2136a8ffe371 WatchSource:0}: Error finding container 060f4a7eebc38180f6438d40611c02ddf1e089fd2397eed0740b2136a8ffe371: Status 404 returned error can't find the container with id 060f4a7eebc38180f6438d40611c02ddf1e089fd2397eed0740b2136a8ffe371
Jul 24 23:33:50.878511 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:50.878481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:51.015150 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.015121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"]
Jul 24 23:33:51.016477 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:33:51.016451    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9303319e_47eb_404e_af84_5d0a2ac92a71.slice/crio-3f0f7bdd409ca3377236284a539e67b2ace010248de679b3c0c4d7102a16f677 WatchSource:0}: Error finding container 3f0f7bdd409ca3377236284a539e67b2ace010248de679b3c0c4d7102a16f677: Status 404 returned error can't find the container with id 3f0f7bdd409ca3377236284a539e67b2ace010248de679b3c0c4d7102a16f677
Jul 24 23:33:51.343778 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.343746    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf" event={"ID":"254fac1f-8d2a-48b4-acfd-96a41c354564","Type":"ContainerStarted","Data":"5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6"}
Jul 24 23:33:51.343995 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.343802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf" event={"ID":"254fac1f-8d2a-48b4-acfd-96a41c354564","Type":"ContainerStarted","Data":"060f4a7eebc38180f6438d40611c02ddf1e089fd2397eed0740b2136a8ffe371"}
Jul 24 23:33:51.343995 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.343833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:33:51.345317 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.345292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd" event={"ID":"9303319e-47eb-404e-af84-5d0a2ac92a71","Type":"ContainerStarted","Data":"b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24"}
Jul 24 23:33:51.345449 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.345335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd" event={"ID":"9303319e-47eb-404e-af84-5d0a2ac92a71","Type":"ContainerStarted","Data":"3f0f7bdd409ca3377236284a539e67b2ace010248de679b3c0c4d7102a16f677"}
Jul 24 23:33:51.345449 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.345373    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:33:51.360765 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.360719    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf" podStartSLOduration=2.0121384 podStartE2EDuration="2.360701996s" totalImagesPullingTime="348.563596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:33:51.360010481 +0000 UTC m=+2847.993289422" watchObservedRunningTime="2026-07-24 23:33:51.360701996 +0000 UTC m=+2847.993980963"
Jul 24 23:33:51.377022 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:51.376973    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd" podStartSLOduration=1.376955962 podStartE2EDuration="1.376955962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:33:51.374948318 +0000 UTC m=+2848.008227273" watchObservedRunningTime="2026-07-24 23:33:51.376955962 +0000 UTC m=+2848.010234901"
Jul 24 23:33:54.191503 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:54.191464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:33:55.133310 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:55.133268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:33:57.356423 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:33:57.356394    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:34:02.353150 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:02.353117    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:34:06.992086 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:06.992054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:22.782409 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:22.782349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:26.726402 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:26.726367    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:34:29.375359 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:29.375315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:42.223260 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:42.223220    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:34:53.041936 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:53.041903    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6_1d71fc1c-a3d8-432f-a474-0894b4a9a06f/payload-processing/0.log"
Jul 24 23:34:53.555599 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:53.555552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:53.863104 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:53.863074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:54.078195 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:54.078144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:56.554884 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.554855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:56.642431 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.642384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:56.644563 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.644539    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"]
Jul 24 23:34:56.648923 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.648901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:56.655208 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.655185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-429acb-serving-cert\""
Jul 24 23:34:56.678160 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.678126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"]
Jul 24 23:34:56.751375 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.751337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wthfh\" (UniqueName: \"kubernetes.io/projected/98334158-d903-4317-8363-c3ddf73e1721-kube-api-access-wthfh\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:56.751595 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.751446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:56.751595 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.751482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98334158-d903-4317-8363-c3ddf73e1721-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:56.759491 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.759455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:56.852932 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.852895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:56.853136 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.852943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98334158-d903-4317-8363-c3ddf73e1721-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:56.853136 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.852980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wthfh\" (UniqueName: \"kubernetes.io/projected/98334158-d903-4317-8363-c3ddf73e1721-kube-api-access-wthfh\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:56.853136 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:34:56.853067    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-serving-cert: secret "maas-api-e2e-ipp-cleanup-429acb-serving-cert" not found
Jul 24 23:34:56.853303 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:34:56.853144    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls podName:98334158-d903-4317-8363-c3ddf73e1721 nodeName:}" failed. No retries permitted until 2026-07-24 23:34:57.353121395 +0000 UTC m=+2913.986400325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc" (UID: "98334158-d903-4317-8363-c3ddf73e1721") : secret "maas-api-e2e-ipp-cleanup-429acb-serving-cert" not found
Jul 24 23:34:56.853602 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.853584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98334158-d903-4317-8363-c3ddf73e1721-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:56.875177 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:56.875150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wthfh\" (UniqueName: \"kubernetes.io/projected/98334158-d903-4317-8363-c3ddf73e1721-kube-api-access-wthfh\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:57.014015 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.013968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:57.357651 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.357616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:57.359993 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.359969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:57.414277 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.414247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:57.527842 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.527810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:34:57.531021 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.530992    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"]
Jul 24 23:34:57.534825 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.534805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:57.540067 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.540042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-429acb-dockercfg-kfh7p\""
Jul 24 23:34:57.541895 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.541876    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-429acb\""
Jul 24 23:34:57.549928 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.549905    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"]
Jul 24 23:34:57.553392 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.553369    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:57.553973 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.553951    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"]
Jul 24 23:34:57.559383 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.559344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:57.559756 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.559672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkfd\" (UniqueName: \"kubernetes.io/projected/4ade3caf-d87a-4e7c-b86a-454700113460-kube-api-access-gwkfd\") pod \"payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5\" (UID: \"4ade3caf-d87a-4e7c-b86a-454700113460\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:57.559819 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.559756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade3caf-d87a-4e7c-b86a-454700113460-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5\" (UID: \"4ade3caf-d87a-4e7c-b86a-454700113460\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:57.581886 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.581852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"]
Jul 24 23:34:57.662008 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.660398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade3caf-d87a-4e7c-b86a-454700113460-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5\" (UID: \"4ade3caf-d87a-4e7c-b86a-454700113460\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:57.662008 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.660547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b09479f-b234-45c0-8614-d14010612b5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6\" (UID: \"4b09479f-b234-45c0-8614-d14010612b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:57.662008 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.660600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkfd\" (UniqueName: \"kubernetes.io/projected/4ade3caf-d87a-4e7c-b86a-454700113460-kube-api-access-gwkfd\") pod \"payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5\" (UID: \"4ade3caf-d87a-4e7c-b86a-454700113460\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:57.662008 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.660637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8qf\" (UniqueName: \"kubernetes.io/projected/4b09479f-b234-45c0-8614-d14010612b5c-kube-api-access-5k8qf\") pod \"payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6\" (UID: \"4b09479f-b234-45c0-8614-d14010612b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:57.662008 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.661567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade3caf-d87a-4e7c-b86a-454700113460-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5\" (UID: \"4ade3caf-d87a-4e7c-b86a-454700113460\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:57.674654 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.674590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkfd\" (UniqueName: \"kubernetes.io/projected/4ade3caf-d87a-4e7c-b86a-454700113460-kube-api-access-gwkfd\") pod \"payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5\" (UID: \"4ade3caf-d87a-4e7c-b86a-454700113460\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:57.728079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.728051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"]
Jul 24 23:34:57.729950 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:34:57.729919    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98334158_d903_4317_8363_c3ddf73e1721.slice/crio-eedb5a81666aba03060993466fb8aa7cc6baf7ec058574a03d9436062474dc65 WatchSource:0}: Error finding container eedb5a81666aba03060993466fb8aa7cc6baf7ec058574a03d9436062474dc65: Status 404 returned error can't find the container with id eedb5a81666aba03060993466fb8aa7cc6baf7ec058574a03d9436062474dc65
Jul 24 23:34:57.761571 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.761510    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b09479f-b234-45c0-8614-d14010612b5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6\" (UID: \"4b09479f-b234-45c0-8614-d14010612b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:57.761755 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.761608    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8qf\" (UniqueName: \"kubernetes.io/projected/4b09479f-b234-45c0-8614-d14010612b5c-kube-api-access-5k8qf\") pod \"payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6\" (UID: \"4b09479f-b234-45c0-8614-d14010612b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:57.762235 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.762207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b09479f-b234-45c0-8614-d14010612b5c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6\" (UID: \"4b09479f-b234-45c0-8614-d14010612b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:57.771066 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.771039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8qf\" (UniqueName: \"kubernetes.io/projected/4b09479f-b234-45c0-8614-d14010612b5c-kube-api-access-5k8qf\") pod \"payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6\" (UID: \"4b09479f-b234-45c0-8614-d14010612b5c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:57.846396 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.846361    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:57.865010 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:57.864978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:58.003088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.003061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"]
Jul 24 23:34:58.005739 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:34:58.005713    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ade3caf_d87a_4e7c_b86a_454700113460.slice/crio-d44a37171e77262c331f4a0384a7a10e92959d29941287c2164d623b14ca9364 WatchSource:0}: Error finding container d44a37171e77262c331f4a0384a7a10e92959d29941287c2164d623b14ca9364: Status 404 returned error can't find the container with id d44a37171e77262c331f4a0384a7a10e92959d29941287c2164d623b14ca9364
Jul 24 23:34:58.020599 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.020567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"]
Jul 24 23:34:58.022741 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:34:58.022705    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b09479f_b234_45c0_8614_d14010612b5c.slice/crio-5568684d876d6b1cfe722e6770c27271076ac3ba60096dd8383f502497ef2c73 WatchSource:0}: Error finding container 5568684d876d6b1cfe722e6770c27271076ac3ba60096dd8383f502497ef2c73: Status 404 returned error can't find the container with id 5568684d876d6b1cfe722e6770c27271076ac3ba60096dd8383f502497ef2c73
Jul 24 23:34:58.683355 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.683310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc" event={"ID":"98334158-d903-4317-8363-c3ddf73e1721","Type":"ContainerStarted","Data":"9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9"}
Jul 24 23:34:58.683815 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.683372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc" event={"ID":"98334158-d903-4317-8363-c3ddf73e1721","Type":"ContainerStarted","Data":"eedb5a81666aba03060993466fb8aa7cc6baf7ec058574a03d9436062474dc65"}
Jul 24 23:34:58.683815 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.683403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:34:58.684979 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.684955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6" event={"ID":"4b09479f-b234-45c0-8614-d14010612b5c","Type":"ContainerStarted","Data":"ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff"}
Jul 24 23:34:58.685065 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.684999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6" event={"ID":"4b09479f-b234-45c0-8614-d14010612b5c","Type":"ContainerStarted","Data":"5568684d876d6b1cfe722e6770c27271076ac3ba60096dd8383f502497ef2c73"}
Jul 24 23:34:58.685065 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.685020    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:34:58.686344 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.686323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5" event={"ID":"4ade3caf-d87a-4e7c-b86a-454700113460","Type":"ContainerStarted","Data":"3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a"}
Jul 24 23:34:58.686405 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.686361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5" event={"ID":"4ade3caf-d87a-4e7c-b86a-454700113460","Type":"ContainerStarted","Data":"d44a37171e77262c331f4a0384a7a10e92959d29941287c2164d623b14ca9364"}
Jul 24 23:34:58.686405 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.686392    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:34:58.703337 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.703285    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc" podStartSLOduration=2.323626127 podStartE2EDuration="2.703265963s" totalImagesPullingTime="379.639836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:58.700798788 +0000 UTC m=+2915.334077739" watchObservedRunningTime="2026-07-24 23:34:58.703265963 +0000 UTC m=+2915.336544903"
Jul 24 23:34:58.719354 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.719300    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6" podStartSLOduration=1.719282396 podStartE2EDuration="1.719282396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:58.71714411 +0000 UTC m=+2915.350423048" watchObservedRunningTime="2026-07-24 23:34:58.719282396 +0000 UTC m=+2915.352561334"
Jul 24 23:34:58.734423 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:34:58.734366    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5" podStartSLOduration=1.734347696 podStartE2EDuration="1.734347696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:58.731904128 +0000 UTC m=+2915.365183090" watchObservedRunningTime="2026-07-24 23:34:58.734347696 +0000 UTC m=+2915.367626639"
Jul 24 23:35:01.059250 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:01.059221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:01.625503 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:01.625467    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:35:04.696531 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:04.696497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:35:09.693733 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:09.693703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:35:09.694222 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:09.693911    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:35:22.384923 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.384888    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"]
Jul 24 23:35:22.385330 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.385162    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc" podUID="98334158-d903-4317-8363-c3ddf73e1721" containerName="maas-api" containerID="cri-o://9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9" gracePeriod=30
Jul 24 23:35:22.420575 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.420540    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"]
Jul 24 23:35:22.420839 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.420803    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6" podUID="4b09479f-b234-45c0-8614-d14010612b5c" containerName="payload-processing" containerID="cri-o://ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff" gracePeriod=30
Jul 24 23:35:22.431269 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.431238    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"]
Jul 24 23:35:22.431890 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.431859    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5" podUID="4ade3caf-d87a-4e7c-b86a-454700113460" containerName="payload-pre-processing" containerID="cri-o://3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a" gracePeriod=30
Jul 24 23:35:22.458258 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.458219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:22.701478 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.701436    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:35:22.747625 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.747603    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:35:22.753904 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.753866    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:35:22.799044 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.799008    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b09479f-b234-45c0-8614-d14010612b5c-plugins-config-volume\") pod \"4b09479f-b234-45c0-8614-d14010612b5c\" (UID: \"4b09479f-b234-45c0-8614-d14010612b5c\") "
Jul 24 23:35:22.799255 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.799062    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k8qf\" (UniqueName: \"kubernetes.io/projected/4b09479f-b234-45c0-8614-d14010612b5c-kube-api-access-5k8qf\") pod \"4b09479f-b234-45c0-8614-d14010612b5c\" (UID: \"4b09479f-b234-45c0-8614-d14010612b5c\") "
Jul 24 23:35:22.799453 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.799429    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b09479f-b234-45c0-8614-d14010612b5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b09479f-b234-45c0-8614-d14010612b5c" (UID: "4b09479f-b234-45c0-8614-d14010612b5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:35:22.801190 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.801160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b09479f-b234-45c0-8614-d14010612b5c-kube-api-access-5k8qf" (OuterVolumeSpecName: "kube-api-access-5k8qf") pod "4b09479f-b234-45c0-8614-d14010612b5c" (UID: "4b09479f-b234-45c0-8614-d14010612b5c"). InnerVolumeSpecName "kube-api-access-5k8qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:35:22.823684 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.823653    2563 generic.go:356] "Generic (PLEG): container finished" podID="98334158-d903-4317-8363-c3ddf73e1721" containerID="9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9" exitCode=0
Jul 24 23:35:22.823865 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.823688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc" event={"ID":"98334158-d903-4317-8363-c3ddf73e1721","Type":"ContainerDied","Data":"9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9"}
Jul 24 23:35:22.823865 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.823708    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"
Jul 24 23:35:22.823865 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.823757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc" event={"ID":"98334158-d903-4317-8363-c3ddf73e1721","Type":"ContainerDied","Data":"eedb5a81666aba03060993466fb8aa7cc6baf7ec058574a03d9436062474dc65"}
Jul 24 23:35:22.823865 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.823788    2563 scope.go:117] "RemoveContainer" containerID="9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9"
Jul 24 23:35:22.824986 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.824966    2563 generic.go:356] "Generic (PLEG): container finished" podID="4b09479f-b234-45c0-8614-d14010612b5c" containerID="ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff" exitCode=0
Jul 24 23:35:22.825076 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.825012    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"
Jul 24 23:35:22.825076 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.825037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6" event={"ID":"4b09479f-b234-45c0-8614-d14010612b5c","Type":"ContainerDied","Data":"ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff"}
Jul 24 23:35:22.825186 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.825076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6" event={"ID":"4b09479f-b234-45c0-8614-d14010612b5c","Type":"ContainerDied","Data":"5568684d876d6b1cfe722e6770c27271076ac3ba60096dd8383f502497ef2c73"}
Jul 24 23:35:22.826493 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.826469    2563 generic.go:356] "Generic (PLEG): container finished" podID="4ade3caf-d87a-4e7c-b86a-454700113460" containerID="3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a" exitCode=0
Jul 24 23:35:22.826630 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.826496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5" event={"ID":"4ade3caf-d87a-4e7c-b86a-454700113460","Type":"ContainerDied","Data":"3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a"}
Jul 24 23:35:22.826630 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.826507    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"
Jul 24 23:35:22.826630 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.826585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5" event={"ID":"4ade3caf-d87a-4e7c-b86a-454700113460","Type":"ContainerDied","Data":"d44a37171e77262c331f4a0384a7a10e92959d29941287c2164d623b14ca9364"}
Jul 24 23:35:22.833122 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.833105    2563 scope.go:117] "RemoveContainer" containerID="9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9"
Jul 24 23:35:22.833410 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:35:22.833390    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9\": container with ID starting with 9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9 not found: ID does not exist" containerID="9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9"
Jul 24 23:35:22.833484 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.833418    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9"} err="failed to get container status \"9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9\": rpc error: code = NotFound desc = could not find container \"9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9\": container with ID starting with 9d5b5e565a44daa28224116bc1533a8d7d66d06948b52dd3bc97039696d51be9 not found: ID does not exist"
Jul 24 23:35:22.833484 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.833435    2563 scope.go:117] "RemoveContainer" containerID="ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff"
Jul 24 23:35:22.841489 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.841470    2563 scope.go:117] "RemoveContainer" containerID="ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff"
Jul 24 23:35:22.841799 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:35:22.841766    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff\": container with ID starting with ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff not found: ID does not exist" containerID="ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff"
Jul 24 23:35:22.841848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.841805    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff"} err="failed to get container status \"ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff\": rpc error: code = NotFound desc = could not find container \"ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff\": container with ID starting with ca2d1a063db9ccc34bfa7c10dba691ea1644376638f2dd690ddad03bc6d70fff not found: ID does not exist"
Jul 24 23:35:22.841848 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.841821    2563 scope.go:117] "RemoveContainer" containerID="3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a"
Jul 24 23:35:22.850305 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.850285    2563 scope.go:117] "RemoveContainer" containerID="3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a"
Jul 24 23:35:22.850620 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:35:22.850596    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a\": container with ID starting with 3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a not found: ID does not exist" containerID="3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a"
Jul 24 23:35:22.850709 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.850628    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a"} err="failed to get container status \"3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a\": rpc error: code = NotFound desc = could not find container \"3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a\": container with ID starting with 3f0a9125d24aff7aeb34937b0bdade3e44b5f742eadd68dff95b097eb814034a not found: ID does not exist"
Jul 24 23:35:22.860771 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.860743    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"]
Jul 24 23:35:22.864576 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.864550    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-429acb-787f6c494-2h2m6"]
Jul 24 23:35:22.899761 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.899714    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98334158-d903-4317-8363-c3ddf73e1721-openshift-service-ca\") pod \"98334158-d903-4317-8363-c3ddf73e1721\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") "
Jul 24 23:35:22.899873 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.899826    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wthfh\" (UniqueName: \"kubernetes.io/projected/98334158-d903-4317-8363-c3ddf73e1721-kube-api-access-wthfh\") pod \"98334158-d903-4317-8363-c3ddf73e1721\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") "
Jul 24 23:35:22.899873 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.899856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls\") pod \"98334158-d903-4317-8363-c3ddf73e1721\" (UID: \"98334158-d903-4317-8363-c3ddf73e1721\") "
Jul 24 23:35:22.899960 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.899889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade3caf-d87a-4e7c-b86a-454700113460-plugins-config-volume\") pod \"4ade3caf-d87a-4e7c-b86a-454700113460\" (UID: \"4ade3caf-d87a-4e7c-b86a-454700113460\") "
Jul 24 23:35:22.900004 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.899970    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwkfd\" (UniqueName: \"kubernetes.io/projected/4ade3caf-d87a-4e7c-b86a-454700113460-kube-api-access-gwkfd\") pod \"4ade3caf-d87a-4e7c-b86a-454700113460\" (UID: \"4ade3caf-d87a-4e7c-b86a-454700113460\") "
Jul 24 23:35:22.900386 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.900225    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8qf\" (UniqueName: \"kubernetes.io/projected/4b09479f-b234-45c0-8614-d14010612b5c-kube-api-access-5k8qf\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:22.900386 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.900231    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98334158-d903-4317-8363-c3ddf73e1721-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98334158-d903-4317-8363-c3ddf73e1721" (UID: "98334158-d903-4317-8363-c3ddf73e1721"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:35:22.900386 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.900246    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b09479f-b234-45c0-8614-d14010612b5c-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:22.900386 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.900324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ade3caf-d87a-4e7c-b86a-454700113460-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ade3caf-d87a-4e7c-b86a-454700113460" (UID: "4ade3caf-d87a-4e7c-b86a-454700113460"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:35:22.902530 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.902492    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98334158-d903-4317-8363-c3ddf73e1721" (UID: "98334158-d903-4317-8363-c3ddf73e1721"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:35:22.902619 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.902495    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98334158-d903-4317-8363-c3ddf73e1721-kube-api-access-wthfh" (OuterVolumeSpecName: "kube-api-access-wthfh") pod "98334158-d903-4317-8363-c3ddf73e1721" (UID: "98334158-d903-4317-8363-c3ddf73e1721"). InnerVolumeSpecName "kube-api-access-wthfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:35:22.902692 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:22.902670    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ade3caf-d87a-4e7c-b86a-454700113460-kube-api-access-gwkfd" (OuterVolumeSpecName: "kube-api-access-gwkfd") pod "4ade3caf-d87a-4e7c-b86a-454700113460" (UID: "4ade3caf-d87a-4e7c-b86a-454700113460"). InnerVolumeSpecName "kube-api-access-gwkfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:35:23.001635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.001554    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wthfh\" (UniqueName: \"kubernetes.io/projected/98334158-d903-4317-8363-c3ddf73e1721-kube-api-access-wthfh\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:23.001635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.001587    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98334158-d903-4317-8363-c3ddf73e1721-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:23.001635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.001600    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ade3caf-d87a-4e7c-b86a-454700113460-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:23.001635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.001610    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwkfd\" (UniqueName: \"kubernetes.io/projected/4ade3caf-d87a-4e7c-b86a-454700113460-kube-api-access-gwkfd\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:23.001635 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.001619    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98334158-d903-4317-8363-c3ddf73e1721-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:23.146223 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.146186    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"]
Jul 24 23:35:23.152459 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.152427    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-429acb-9f5fddb75-2ftjc"]
Jul 24 23:35:23.162683 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.162651    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"]
Jul 24 23:35:23.170632 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.170603    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-429acb-6cd5d44f44-kszf5"]
Jul 24 23:35:23.842685 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.842662    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ade3caf-d87a-4e7c-b86a-454700113460" path="/var/lib/kubelet/pods/4ade3caf-d87a-4e7c-b86a-454700113460/volumes"
Jul 24 23:35:23.843027 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.843000    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b09479f-b234-45c0-8614-d14010612b5c" path="/var/lib/kubelet/pods/4b09479f-b234-45c0-8614-d14010612b5c/volumes"
Jul 24 23:35:23.843333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:23.843320    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98334158-d903-4317-8363-c3ddf73e1721" path="/var/lib/kubelet/pods/98334158-d903-4317-8363-c3ddf73e1721/volumes"
Jul 24 23:35:24.267025 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:24.266940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:27.046097 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:27.046065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:27.450421 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:27.450389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:28.425289 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:28.425247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:35:33.134144 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:33.134106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:33.185926 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:33.185898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:33.389325 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:33.389235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:33.608380 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:33.608346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:36.976642 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.976605    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"]
Jul 24 23:35:36.977032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977006    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ade3caf-d87a-4e7c-b86a-454700113460" containerName="payload-pre-processing"
Jul 24 23:35:36.977032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977017    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ade3caf-d87a-4e7c-b86a-454700113460" containerName="payload-pre-processing"
Jul 24 23:35:36.977106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977036    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b09479f-b234-45c0-8614-d14010612b5c" containerName="payload-processing"
Jul 24 23:35:36.977106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977041    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b09479f-b234-45c0-8614-d14010612b5c" containerName="payload-processing"
Jul 24 23:35:36.977106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977057    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98334158-d903-4317-8363-c3ddf73e1721" containerName="maas-api"
Jul 24 23:35:36.977106 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977062    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="98334158-d903-4317-8363-c3ddf73e1721" containerName="maas-api"
Jul 24 23:35:36.977229 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977112    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="98334158-d903-4317-8363-c3ddf73e1721" containerName="maas-api"
Jul 24 23:35:36.977229 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977120    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ade3caf-d87a-4e7c-b86a-454700113460" containerName="payload-pre-processing"
Jul 24 23:35:36.977229 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.977127    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b09479f-b234-45c0-8614-d14010612b5c" containerName="payload-processing"
Jul 24 23:35:36.979967 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.979950    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"
Jul 24 23:35:36.982685 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.982662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:35:36.986999 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:36.986974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"]
Jul 24 23:35:37.131651 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:37.131613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2664e1-937d-4f35-8681-823bb480125e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69\" (UID: \"ce2664e1-937d-4f35-8681-823bb480125e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"
Jul 24 23:35:37.232600 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:37.232467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2664e1-937d-4f35-8681-823bb480125e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69\" (UID: \"ce2664e1-937d-4f35-8681-823bb480125e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"
Jul 24 23:35:37.233098 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:37.233078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2664e1-937d-4f35-8681-823bb480125e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69\" (UID: \"ce2664e1-937d-4f35-8681-823bb480125e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"
Jul 24 23:35:37.291573 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:37.291515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"
Jul 24 23:35:37.427955 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:37.427919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"]
Jul 24 23:35:37.428314 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:35:37.428280    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2664e1_937d_4f35_8681_823bb480125e.slice/crio-644ef5a61eb6ab977e9d3668d4ea0273f5ccc4764f844c71d92fd401c7eadce1 WatchSource:0}: Error finding container 644ef5a61eb6ab977e9d3668d4ea0273f5ccc4764f844c71d92fd401c7eadce1: Status 404 returned error can't find the container with id 644ef5a61eb6ab977e9d3668d4ea0273f5ccc4764f844c71d92fd401c7eadce1
Jul 24 23:35:37.907087 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:37.907051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69" event={"ID":"ce2664e1-937d-4f35-8681-823bb480125e","Type":"ContainerStarted","Data":"7f3824921ebae0d007f1c91cb06f4c847f44a87280c1aff51f144e99bdcf44c1"}
Jul 24 23:35:37.907276 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:37.907103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69" event={"ID":"ce2664e1-937d-4f35-8681-823bb480125e","Type":"ContainerStarted","Data":"644ef5a61eb6ab977e9d3668d4ea0273f5ccc4764f844c71d92fd401c7eadce1"}
Jul 24 23:35:37.923197 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:37.923145    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69" podStartSLOduration=1.9231296759999998 podStartE2EDuration="1.923129676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:35:37.921599081 +0000 UTC m=+2954.554878020" watchObservedRunningTime="2026-07-24 23:35:37.923129676 +0000 UTC m=+2954.556408618"
Jul 24 23:35:38.913655 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:38.913619    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce2664e1-937d-4f35-8681-823bb480125e" containerID="7f3824921ebae0d007f1c91cb06f4c847f44a87280c1aff51f144e99bdcf44c1" exitCode=0
Jul 24 23:35:38.914049 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:38.913708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69" event={"ID":"ce2664e1-937d-4f35-8681-823bb480125e","Type":"ContainerDied","Data":"7f3824921ebae0d007f1c91cb06f4c847f44a87280c1aff51f144e99bdcf44c1"}
Jul 24 23:35:40.058572 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:40.058546    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"
Jul 24 23:35:40.156857 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:40.156818    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2664e1-937d-4f35-8681-823bb480125e-maas-api-service-ca\") pod \"ce2664e1-937d-4f35-8681-823bb480125e\" (UID: \"ce2664e1-937d-4f35-8681-823bb480125e\") "
Jul 24 23:35:40.157167 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:40.157142    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2664e1-937d-4f35-8681-823bb480125e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce2664e1-937d-4f35-8681-823bb480125e" (UID: "ce2664e1-937d-4f35-8681-823bb480125e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:35:40.258302 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:40.258223    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce2664e1-937d-4f35-8681-823bb480125e-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:40.925129 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:40.925096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69" event={"ID":"ce2664e1-937d-4f35-8681-823bb480125e","Type":"ContainerDied","Data":"644ef5a61eb6ab977e9d3668d4ea0273f5ccc4764f844c71d92fd401c7eadce1"}
Jul 24 23:35:40.925129 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:40.925116    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"
Jul 24 23:35:40.925339 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:40.925145    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644ef5a61eb6ab977e9d3668d4ea0273f5ccc4764f844c71d92fd401c7eadce1"
Jul 24 23:35:51.231956 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:51.231868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:35:52.419888 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.419855    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"]
Jul 24 23:35:52.420305 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.420063    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs" podUID="90031a1f-a8a1-4453-a921-f5ca99b9d017" containerName="maas-api" containerID="cri-o://719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225" gracePeriod=30
Jul 24 23:35:52.439301 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.439273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:52.460624 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.460585    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"]
Jul 24 23:35:52.460873 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.460845    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6" podUID="1d71fc1c-a3d8-432f-a474-0894b4a9a06f" containerName="payload-processing" containerID="cri-o://2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311" gracePeriod=30
Jul 24 23:35:52.469296 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.469263    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"]
Jul 24 23:35:52.470549 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.469479    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw" podUID="42727336-91de-48e1-ae90-7e867acddb13" containerName="payload-pre-processing" containerID="cri-o://cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197" gracePeriod=30
Jul 24 23:35:52.502462 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.502428    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:52.561535 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.561488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:52.736205 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.736173    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:35:52.762752 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.762719    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:35:52.767159 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.767137    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ms66\" (UniqueName: \"kubernetes.io/projected/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-kube-api-access-5ms66\") pod \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\" (UID: \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\") "
Jul 24 23:35:52.767293 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.767277    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-plugins-config-volume\") pod \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\" (UID: \"1d71fc1c-a3d8-432f-a474-0894b4a9a06f\") "
Jul 24 23:35:52.767769 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.767741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d71fc1c-a3d8-432f-a474-0894b4a9a06f" (UID: "1d71fc1c-a3d8-432f-a474-0894b4a9a06f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:35:52.769256 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.769234    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-kube-api-access-5ms66" (OuterVolumeSpecName: "kube-api-access-5ms66") pod "1d71fc1c-a3d8-432f-a474-0894b4a9a06f" (UID: "1d71fc1c-a3d8-432f-a474-0894b4a9a06f"). InnerVolumeSpecName "kube-api-access-5ms66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:35:52.775176 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.775155    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:35:52.868227 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.868189    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42727336-91de-48e1-ae90-7e867acddb13-plugins-config-volume\") pod \"42727336-91de-48e1-ae90-7e867acddb13\" (UID: \"42727336-91de-48e1-ae90-7e867acddb13\") "
Jul 24 23:35:52.868407 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.868283    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls\") pod \"90031a1f-a8a1-4453-a921-f5ca99b9d017\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") "
Jul 24 23:35:52.868459 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.868436    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9khr\" (UniqueName: \"kubernetes.io/projected/42727336-91de-48e1-ae90-7e867acddb13-kube-api-access-x9khr\") pod \"42727336-91de-48e1-ae90-7e867acddb13\" (UID: \"42727336-91de-48e1-ae90-7e867acddb13\") "
Jul 24 23:35:52.868506 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.868476    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90031a1f-a8a1-4453-a921-f5ca99b9d017-openshift-service-ca\") pod \"90031a1f-a8a1-4453-a921-f5ca99b9d017\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") "
Jul 24 23:35:52.868610 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.868515    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fljgl\" (UniqueName: \"kubernetes.io/projected/90031a1f-a8a1-4453-a921-f5ca99b9d017-kube-api-access-fljgl\") pod \"90031a1f-a8a1-4453-a921-f5ca99b9d017\" (UID: \"90031a1f-a8a1-4453-a921-f5ca99b9d017\") "
Jul 24 23:35:52.868672 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.868646    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42727336-91de-48e1-ae90-7e867acddb13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42727336-91de-48e1-ae90-7e867acddb13" (UID: "42727336-91de-48e1-ae90-7e867acddb13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:35:52.868875 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.868850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90031a1f-a8a1-4453-a921-f5ca99b9d017-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90031a1f-a8a1-4453-a921-f5ca99b9d017" (UID: "90031a1f-a8a1-4453-a921-f5ca99b9d017"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:35:52.868995 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.868976    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ms66\" (UniqueName: \"kubernetes.io/projected/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-kube-api-access-5ms66\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:52.869062 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.869000    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90031a1f-a8a1-4453-a921-f5ca99b9d017-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:52.869062 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.869012    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d71fc1c-a3d8-432f-a474-0894b4a9a06f-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:52.869062 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.869025    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42727336-91de-48e1-ae90-7e867acddb13-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:52.870369 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.870348    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90031a1f-a8a1-4453-a921-f5ca99b9d017" (UID: "90031a1f-a8a1-4453-a921-f5ca99b9d017"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:35:52.870475 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.870454    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90031a1f-a8a1-4453-a921-f5ca99b9d017-kube-api-access-fljgl" (OuterVolumeSpecName: "kube-api-access-fljgl") pod "90031a1f-a8a1-4453-a921-f5ca99b9d017" (UID: "90031a1f-a8a1-4453-a921-f5ca99b9d017"). InnerVolumeSpecName "kube-api-access-fljgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:35:52.870514 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.870477    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42727336-91de-48e1-ae90-7e867acddb13-kube-api-access-x9khr" (OuterVolumeSpecName: "kube-api-access-x9khr") pod "42727336-91de-48e1-ae90-7e867acddb13" (UID: "42727336-91de-48e1-ae90-7e867acddb13"). InnerVolumeSpecName "kube-api-access-x9khr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:35:52.970374 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.970286    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fljgl\" (UniqueName: \"kubernetes.io/projected/90031a1f-a8a1-4453-a921-f5ca99b9d017-kube-api-access-fljgl\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:52.970374 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.970316    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90031a1f-a8a1-4453-a921-f5ca99b9d017-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:52.970374 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.970329    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9khr\" (UniqueName: \"kubernetes.io/projected/42727336-91de-48e1-ae90-7e867acddb13-kube-api-access-x9khr\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:35:52.984345 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.984309    2563 generic.go:356] "Generic (PLEG): container finished" podID="42727336-91de-48e1-ae90-7e867acddb13" containerID="cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197" exitCode=0
Jul 24 23:35:52.984530 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.984352    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"
Jul 24 23:35:52.984530 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.984393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw" event={"ID":"42727336-91de-48e1-ae90-7e867acddb13","Type":"ContainerDied","Data":"cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197"}
Jul 24 23:35:52.984530 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.984445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw" event={"ID":"42727336-91de-48e1-ae90-7e867acddb13","Type":"ContainerDied","Data":"c3d11ee3ef52bf1b1f591add00d574b6bc37a09102d93f347afbf64fe50eebf8"}
Jul 24 23:35:52.984530 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.984477    2563 scope.go:117] "RemoveContainer" containerID="cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197"
Jul 24 23:35:52.985678 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.985655    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d71fc1c-a3d8-432f-a474-0894b4a9a06f" containerID="2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311" exitCode=0
Jul 24 23:35:52.985764 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.985683    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6" event={"ID":"1d71fc1c-a3d8-432f-a474-0894b4a9a06f","Type":"ContainerDied","Data":"2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311"}
Jul 24 23:35:52.985764 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.985703    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"
Jul 24 23:35:52.985764 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.985722    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6" event={"ID":"1d71fc1c-a3d8-432f-a474-0894b4a9a06f","Type":"ContainerDied","Data":"e13c9f2c7d28bb0158a64af9415c48f6c4295580745caa9499f092a73ff175e0"}
Jul 24 23:35:52.987020 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.987001    2563 generic.go:356] "Generic (PLEG): container finished" podID="90031a1f-a8a1-4453-a921-f5ca99b9d017" containerID="719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225" exitCode=0
Jul 24 23:35:52.987122 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.987057    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"
Jul 24 23:35:52.987122 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.987082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs" event={"ID":"90031a1f-a8a1-4453-a921-f5ca99b9d017","Type":"ContainerDied","Data":"719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225"}
Jul 24 23:35:52.987203 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.987124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs" event={"ID":"90031a1f-a8a1-4453-a921-f5ca99b9d017","Type":"ContainerDied","Data":"4a8e8ab956a447490701b083ec30d956ef9e57f10977bb9953573aea440c5c1b"}
Jul 24 23:35:52.995862 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.995675    2563 scope.go:117] "RemoveContainer" containerID="cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197"
Jul 24 23:35:52.996728 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:35:52.996701    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197\": container with ID starting with cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197 not found: ID does not exist" containerID="cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197"
Jul 24 23:35:52.996859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.996742    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197"} err="failed to get container status \"cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197\": rpc error: code = NotFound desc = could not find container \"cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197\": container with ID starting with cb3e33b871601ab7fd8367f421444d32053283769d3d14cacf29c0a27b080197 not found: ID does not exist"
Jul 24 23:35:52.996859 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:52.996764    2563 scope.go:117] "RemoveContainer" containerID="2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311"
Jul 24 23:35:53.007742 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.007724    2563 scope.go:117] "RemoveContainer" containerID="2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311"
Jul 24 23:35:53.008091 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:35:53.008067    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311\": container with ID starting with 2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311 not found: ID does not exist" containerID="2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311"
Jul 24 23:35:53.008193 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.008097    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311"} err="failed to get container status \"2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311\": rpc error: code = NotFound desc = could not find container \"2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311\": container with ID starting with 2444b415443675755c093c2949615e72a68c11d64ae4a70c258fef4878647311 not found: ID does not exist"
Jul 24 23:35:53.008193 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.008114    2563 scope.go:117] "RemoveContainer" containerID="719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225"
Jul 24 23:35:53.014373 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.014349    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"]
Jul 24 23:35:53.016800 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.016784    2563 scope.go:117] "RemoveContainer" containerID="719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225"
Jul 24 23:35:53.017104 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:35:53.017085    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225\": container with ID starting with 719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225 not found: ID does not exist" containerID="719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225"
Jul 24 23:35:53.017153 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.017113    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225"} err="failed to get container status \"719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225\": rpc error: code = NotFound desc = could not find container \"719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225\": container with ID starting with 719de3ae1b48e7d8921f6f4d6bda23a984cc35aea5143e9b3d15f78051b7f225 not found: ID does not exist"
Jul 24 23:35:53.019866 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.019841    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eb4c8d-7ddf9597b-95qzw"]
Jul 24 23:35:53.034547 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.030986    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"]
Jul 24 23:35:53.037270 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.036202    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eb4c8d-7768467849-b7ggs"]
Jul 24 23:35:53.046738 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.046707    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"]
Jul 24 23:35:53.048763 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.048737    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb4c8d-7fd867fc54-z9vj6"]
Jul 24 23:35:53.842278 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.842248    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d71fc1c-a3d8-432f-a474-0894b4a9a06f" path="/var/lib/kubelet/pods/1d71fc1c-a3d8-432f-a474-0894b4a9a06f/volumes"
Jul 24 23:35:53.842666 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.842618    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42727336-91de-48e1-ae90-7e867acddb13" path="/var/lib/kubelet/pods/42727336-91de-48e1-ae90-7e867acddb13/volumes"
Jul 24 23:35:53.842949 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:53.842936    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90031a1f-a8a1-4453-a921-f5ca99b9d017" path="/var/lib/kubelet/pods/90031a1f-a8a1-4453-a921-f5ca99b9d017/volumes"
Jul 24 23:35:57.054482 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:57.054448    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"]
Jul 24 23:35:57.054902 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:57.054550    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:57.057932 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:57.057907    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-eb4c8d-f8t69"]
Jul 24 23:35:57.449645 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:57.449614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:35:57.842122 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:35:57.842092    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce2664e1-937d-4f35-8681-823bb480125e" path="/var/lib/kubelet/pods/ce2664e1-937d-4f35-8681-823bb480125e/volumes"
Jul 24 23:36:02.023433 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:02.023398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:36:02.535413 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:02.535385    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:36:23.975098 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:23.975069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:36:23.980966 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:23.980941    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sqlz6_4d27a0bc-41c6-480e-a4e9-9c0386da97f0/ovn-acl-logging/0.log"
Jul 24 23:36:38.583017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:38.582984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:36:39.319099 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:39.319067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:36:44.893463 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:44.893424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:36:50.300956 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.300919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:36:50.400156 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.400128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:36:50.592970 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.592933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:36:50.830608 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.830568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:36:50.951900 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.951815    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"]
Jul 24 23:36:50.952200 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952186    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42727336-91de-48e1-ae90-7e867acddb13" containerName="payload-pre-processing"
Jul 24 23:36:50.952246 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952201    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42727336-91de-48e1-ae90-7e867acddb13" containerName="payload-pre-processing"
Jul 24 23:36:50.952246 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952213    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90031a1f-a8a1-4453-a921-f5ca99b9d017" containerName="maas-api"
Jul 24 23:36:50.952246 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952219    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="90031a1f-a8a1-4453-a921-f5ca99b9d017" containerName="maas-api"
Jul 24 23:36:50.952246 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952231    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce2664e1-937d-4f35-8681-823bb480125e" containerName="revoke-keys"
Jul 24 23:36:50.952246 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952236    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce2664e1-937d-4f35-8681-823bb480125e" containerName="revoke-keys"
Jul 24 23:36:50.952246 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952244    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d71fc1c-a3d8-432f-a474-0894b4a9a06f" containerName="payload-processing"
Jul 24 23:36:50.952246 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952249    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d71fc1c-a3d8-432f-a474-0894b4a9a06f" containerName="payload-processing"
Jul 24 23:36:50.952461 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952308    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce2664e1-937d-4f35-8681-823bb480125e" containerName="revoke-keys"
Jul 24 23:36:50.952461 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952315    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="90031a1f-a8a1-4453-a921-f5ca99b9d017" containerName="maas-api"
Jul 24 23:36:50.952461 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952321    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42727336-91de-48e1-ae90-7e867acddb13" containerName="payload-pre-processing"
Jul 24 23:36:50.952461 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.952329    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d71fc1c-a3d8-432f-a474-0894b4a9a06f" containerName="payload-processing"
Jul 24 23:36:50.955370 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.955353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"
Jul 24 23:36:50.957847 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.957823    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:36:50.961936 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.961908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"]
Jul 24 23:36:50.980720 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:50.980686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc299e-1570-4b8d-845a-0a6d67082439-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk\" (UID: \"61dc299e-1570-4b8d-845a-0a6d67082439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"
Jul 24 23:36:51.081802 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:51.081765    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc299e-1570-4b8d-845a-0a6d67082439-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk\" (UID: \"61dc299e-1570-4b8d-845a-0a6d67082439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"
Jul 24 23:36:51.082411 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:51.082391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc299e-1570-4b8d-845a-0a6d67082439-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk\" (UID: \"61dc299e-1570-4b8d-845a-0a6d67082439\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"
Jul 24 23:36:51.267074 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:51.266982    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"
Jul 24 23:36:51.409475 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:51.409449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"]
Jul 24 23:36:51.411549 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:36:51.411485    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61dc299e_1570_4b8d_845a_0a6d67082439.slice/crio-16ffe5e8c0fb1d03b38f693979f42b0662b9cc7cc02ece6e8ab09424ffd77459 WatchSource:0}: Error finding container 16ffe5e8c0fb1d03b38f693979f42b0662b9cc7cc02ece6e8ab09424ffd77459: Status 404 returned error can't find the container with id 16ffe5e8c0fb1d03b38f693979f42b0662b9cc7cc02ece6e8ab09424ffd77459
Jul 24 23:36:52.305054 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:52.305015    2563 generic.go:356] "Generic (PLEG): container finished" podID="61dc299e-1570-4b8d-845a-0a6d67082439" containerID="699483cdf1235b9ff1be0ebfe992de448323dc7b89dd83d1a570ca6742836354" exitCode=0
Jul 24 23:36:52.305280 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:52.305098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk" event={"ID":"61dc299e-1570-4b8d-845a-0a6d67082439","Type":"ContainerDied","Data":"699483cdf1235b9ff1be0ebfe992de448323dc7b89dd83d1a570ca6742836354"}
Jul 24 23:36:52.305280 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:52.305161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk" event={"ID":"61dc299e-1570-4b8d-845a-0a6d67082439","Type":"ContainerStarted","Data":"16ffe5e8c0fb1d03b38f693979f42b0662b9cc7cc02ece6e8ab09424ffd77459"}
Jul 24 23:36:53.452271 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:53.452242    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"
Jul 24 23:36:53.502825 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:53.502788    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc299e-1570-4b8d-845a-0a6d67082439-maas-api-service-ca\") pod \"61dc299e-1570-4b8d-845a-0a6d67082439\" (UID: \"61dc299e-1570-4b8d-845a-0a6d67082439\") "
Jul 24 23:36:53.503191 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:53.503167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dc299e-1570-4b8d-845a-0a6d67082439-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61dc299e-1570-4b8d-845a-0a6d67082439" (UID: "61dc299e-1570-4b8d-845a-0a6d67082439"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:36:53.604556 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:53.604477    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dc299e-1570-4b8d-845a-0a6d67082439-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:36:54.317830 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:54.317803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"
Jul 24 23:36:54.318047 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:54.317824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk" event={"ID":"61dc299e-1570-4b8d-845a-0a6d67082439","Type":"ContainerDied","Data":"16ffe5e8c0fb1d03b38f693979f42b0662b9cc7cc02ece6e8ab09424ffd77459"}
Jul 24 23:36:54.318047 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:36:54.317881    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16ffe5e8c0fb1d03b38f693979f42b0662b9cc7cc02ece6e8ab09424ffd77459"
Jul 24 23:37:01.344325 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.344273    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"]
Jul 24 23:37:01.344835 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.344571    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf" podUID="254fac1f-8d2a-48b4-acfd-96a41c354564" containerName="maas-api" containerID="cri-o://5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6" gracePeriod=30
Jul 24 23:37:01.367082 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.367055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:01.403129 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.403098    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"]
Jul 24 23:37:01.403394 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.403365    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd" podUID="9303319e-47eb-404e-af84-5d0a2ac92a71" containerName="payload-pre-processing" containerID="cri-o://b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24" gracePeriod=30
Jul 24 23:37:01.431329 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.431298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:01.531899 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.531864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:01.657278 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.657250    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:37:01.694397 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.694363    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:37:01.784047 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.784011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9303319e-47eb-404e-af84-5d0a2ac92a71-plugins-config-volume\") pod \"9303319e-47eb-404e-af84-5d0a2ac92a71\" (UID: \"9303319e-47eb-404e-af84-5d0a2ac92a71\") "
Jul 24 23:37:01.784258 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.784069    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp79j\" (UniqueName: \"kubernetes.io/projected/9303319e-47eb-404e-af84-5d0a2ac92a71-kube-api-access-bp79j\") pod \"9303319e-47eb-404e-af84-5d0a2ac92a71\" (UID: \"9303319e-47eb-404e-af84-5d0a2ac92a71\") "
Jul 24 23:37:01.784258 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.784122    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fac1f-8d2a-48b4-acfd-96a41c354564-openshift-service-ca\") pod \"254fac1f-8d2a-48b4-acfd-96a41c354564\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") "
Jul 24 23:37:01.784258 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.784152    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvdsz\" (UniqueName: \"kubernetes.io/projected/254fac1f-8d2a-48b4-acfd-96a41c354564-kube-api-access-cvdsz\") pod \"254fac1f-8d2a-48b4-acfd-96a41c354564\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") "
Jul 24 23:37:01.784258 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.784192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls\") pod \"254fac1f-8d2a-48b4-acfd-96a41c354564\" (UID: \"254fac1f-8d2a-48b4-acfd-96a41c354564\") "
Jul 24 23:37:01.784492 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.784462    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9303319e-47eb-404e-af84-5d0a2ac92a71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9303319e-47eb-404e-af84-5d0a2ac92a71" (UID: "9303319e-47eb-404e-af84-5d0a2ac92a71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:01.784599 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.784552    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254fac1f-8d2a-48b4-acfd-96a41c354564-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "254fac1f-8d2a-48b4-acfd-96a41c354564" (UID: "254fac1f-8d2a-48b4-acfd-96a41c354564"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:01.786345 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.786315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "254fac1f-8d2a-48b4-acfd-96a41c354564" (UID: "254fac1f-8d2a-48b4-acfd-96a41c354564"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:37:01.786439 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.786371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9303319e-47eb-404e-af84-5d0a2ac92a71-kube-api-access-bp79j" (OuterVolumeSpecName: "kube-api-access-bp79j") pod "9303319e-47eb-404e-af84-5d0a2ac92a71" (UID: "9303319e-47eb-404e-af84-5d0a2ac92a71"). InnerVolumeSpecName "kube-api-access-bp79j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:37:01.786439 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.786405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254fac1f-8d2a-48b4-acfd-96a41c354564-kube-api-access-cvdsz" (OuterVolumeSpecName: "kube-api-access-cvdsz") pod "254fac1f-8d2a-48b4-acfd-96a41c354564" (UID: "254fac1f-8d2a-48b4-acfd-96a41c354564"). InnerVolumeSpecName "kube-api-access-cvdsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:37:01.885663 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.885572    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/254fac1f-8d2a-48b4-acfd-96a41c354564-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:01.885663 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.885604    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvdsz\" (UniqueName: \"kubernetes.io/projected/254fac1f-8d2a-48b4-acfd-96a41c354564-kube-api-access-cvdsz\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:01.885663 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.885614    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/254fac1f-8d2a-48b4-acfd-96a41c354564-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:01.885663 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.885623    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9303319e-47eb-404e-af84-5d0a2ac92a71-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:01.885663 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:01.885632    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp79j\" (UniqueName: \"kubernetes.io/projected/9303319e-47eb-404e-af84-5d0a2ac92a71-kube-api-access-bp79j\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:02.368100 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.368069    2563 generic.go:356] "Generic (PLEG): container finished" podID="254fac1f-8d2a-48b4-acfd-96a41c354564" containerID="5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6" exitCode=0
Jul 24 23:37:02.368554 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.368109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf" event={"ID":"254fac1f-8d2a-48b4-acfd-96a41c354564","Type":"ContainerDied","Data":"5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6"}
Jul 24 23:37:02.368554 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.368135    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"
Jul 24 23:37:02.368554 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.368176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf" event={"ID":"254fac1f-8d2a-48b4-acfd-96a41c354564","Type":"ContainerDied","Data":"060f4a7eebc38180f6438d40611c02ddf1e089fd2397eed0740b2136a8ffe371"}
Jul 24 23:37:02.368554 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.368206    2563 scope.go:117] "RemoveContainer" containerID="5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6"
Jul 24 23:37:02.369748 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.369727    2563 generic.go:356] "Generic (PLEG): container finished" podID="9303319e-47eb-404e-af84-5d0a2ac92a71" containerID="b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24" exitCode=0
Jul 24 23:37:02.369850 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.369833    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"
Jul 24 23:37:02.369908 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.369887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd" event={"ID":"9303319e-47eb-404e-af84-5d0a2ac92a71","Type":"ContainerDied","Data":"b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24"}
Jul 24 23:37:02.369964 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.369934    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd" event={"ID":"9303319e-47eb-404e-af84-5d0a2ac92a71","Type":"ContainerDied","Data":"3f0f7bdd409ca3377236284a539e67b2ace010248de679b3c0c4d7102a16f677"}
Jul 24 23:37:02.377699 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.377678    2563 scope.go:117] "RemoveContainer" containerID="5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6"
Jul 24 23:37:02.377968 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:37:02.377950    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6\": container with ID starting with 5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6 not found: ID does not exist" containerID="5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6"
Jul 24 23:37:02.378037 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.377977    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6"} err="failed to get container status \"5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6\": rpc error: code = NotFound desc = could not find container \"5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6\": container with ID starting with 5aa4a405c74f8656c99007a7dd14a64abbd9b6679d3da11058ef58dab0f6eee6 not found: ID does not exist"
Jul 24 23:37:02.378037 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.378000    2563 scope.go:117] "RemoveContainer" containerID="b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24"
Jul 24 23:37:02.385832 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.385808    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"]
Jul 24 23:37:02.386443 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.386425    2563 scope.go:117] "RemoveContainer" containerID="b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24"
Jul 24 23:37:02.386739 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:37:02.386721    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24\": container with ID starting with b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24 not found: ID does not exist" containerID="b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24"
Jul 24 23:37:02.386810 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.386749    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24"} err="failed to get container status \"b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24\": rpc error: code = NotFound desc = could not find container \"b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24\": container with ID starting with b21265895c0159d1e3506322a8210853206f8348ceae601674127fa05bf9ff24 not found: ID does not exist"
Jul 24 23:37:02.390061 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.390037    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-064932-7bffb4cd56-ljzjf"]
Jul 24 23:37:02.399751 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.399723    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"]
Jul 24 23:37:02.402156 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:02.402133    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-064932-559446b548-s2krd"]
Jul 24 23:37:03.841419 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:03.841388    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="254fac1f-8d2a-48b4-acfd-96a41c354564" path="/var/lib/kubelet/pods/254fac1f-8d2a-48b4-acfd-96a41c354564/volumes"
Jul 24 23:37:03.841829 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:03.841752    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9303319e-47eb-404e-af84-5d0a2ac92a71" path="/var/lib/kubelet/pods/9303319e-47eb-404e-af84-5d0a2ac92a71/volumes"
Jul 24 23:37:04.527378 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:04.527342    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:37:06.021305 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:06.021247    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"]
Jul 24 23:37:06.023310 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:06.023283    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-064932-tg5jk"]
Jul 24 23:37:06.023471 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:06.023420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:06.424670 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:06.424620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:07.842041 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:07.842007    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61dc299e-1570-4b8d-845a-0a6d67082439" path="/var/lib/kubelet/pods/61dc299e-1570-4b8d-845a-0a6d67082439/volumes"
Jul 24 23:37:12.227425 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:12.227386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:12.285937 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:12.285905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:12.499966 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:12.499895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:12.720665 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:12.720635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:23.264894 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.264859    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"]
Jul 24 23:37:23.265302 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.265062    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs" podUID="144d2566-8c25-419b-930f-c64fa9def932" containerName="maas-api" containerID="cri-o://5d86b7634e29deab0fef7c7b64b538a3ab7a29b7e139c17fa4a8769c4d5a9b9f" gracePeriod=30
Jul 24 23:37:23.283299 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.283272    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"]
Jul 24 23:37:23.290333 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.290304    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5c850d-29748930-vg28w"]
Jul 24 23:37:23.292791 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.292755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:23.311599 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.311568    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"]
Jul 24 23:37:23.311798 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.311777    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c" podUID="9c886d69-8bb4-4e02-b966-aa28d31d36dc" containerName="payload-processing" containerID="cri-o://a037817cae8d8678feea24f87ac9171ecd57ebb12170362a478d62f16d3a0702" gracePeriod=30
Jul 24 23:37:23.313801 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.313751    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs" podUID="144d2566-8c25-419b-930f-c64fa9def932" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.71:8443/health\": dial tcp 10.134.0.71:8443: connect: connection refused"
Jul 24 23:37:23.320264 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.320171    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"]
Jul 24 23:37:23.320454 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.320414    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j" podUID="d19efbd9-82bf-4a27-8f9a-c542633bbac3" containerName="payload-pre-processing" containerID="cri-o://9f643234b81ebcafdca5f44076f5f88ae146e55ce811a681367f28646edeadfb" gracePeriod=30
Jul 24 23:37:23.392449 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.392422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:23.445476 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.445375    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:23.482975 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.482794    2563 generic.go:356] "Generic (PLEG): container finished" podID="9c886d69-8bb4-4e02-b966-aa28d31d36dc" containerID="a037817cae8d8678feea24f87ac9171ecd57ebb12170362a478d62f16d3a0702" exitCode=0
Jul 24 23:37:23.482975 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.482839    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c" event={"ID":"9c886d69-8bb4-4e02-b966-aa28d31d36dc","Type":"ContainerDied","Data":"a037817cae8d8678feea24f87ac9171ecd57ebb12170362a478d62f16d3a0702"}
Jul 24 23:37:23.485325 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.485136    2563 generic.go:356] "Generic (PLEG): container finished" podID="d19efbd9-82bf-4a27-8f9a-c542633bbac3" containerID="9f643234b81ebcafdca5f44076f5f88ae146e55ce811a681367f28646edeadfb" exitCode=0
Jul 24 23:37:23.485325 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.485212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j" event={"ID":"d19efbd9-82bf-4a27-8f9a-c542633bbac3","Type":"ContainerDied","Data":"9f643234b81ebcafdca5f44076f5f88ae146e55ce811a681367f28646edeadfb"}
Jul 24 23:37:23.486930 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.486896    2563 generic.go:356] "Generic (PLEG): container finished" podID="144d2566-8c25-419b-930f-c64fa9def932" containerID="5d86b7634e29deab0fef7c7b64b538a3ab7a29b7e139c17fa4a8769c4d5a9b9f" exitCode=0
Jul 24 23:37:23.487057 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.486958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs" event={"ID":"144d2566-8c25-419b-930f-c64fa9def932","Type":"ContainerDied","Data":"5d86b7634e29deab0fef7c7b64b538a3ab7a29b7e139c17fa4a8769c4d5a9b9f"}
Jul 24 23:37:23.575490 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.575461    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:37:23.593568 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.592638    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:37:23.593568 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.593256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vp9s\" (UniqueName: \"kubernetes.io/projected/9c886d69-8bb4-4e02-b966-aa28d31d36dc-kube-api-access-7vp9s\") pod \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\" (UID: \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\") "
Jul 24 23:37:23.593568 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.593325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c886d69-8bb4-4e02-b966-aa28d31d36dc-plugins-config-volume\") pod \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\" (UID: \"9c886d69-8bb4-4e02-b966-aa28d31d36dc\") "
Jul 24 23:37:23.593897 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.593870    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c886d69-8bb4-4e02-b966-aa28d31d36dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c886d69-8bb4-4e02-b966-aa28d31d36dc" (UID: "9c886d69-8bb4-4e02-b966-aa28d31d36dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:23.597124 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.597096    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c886d69-8bb4-4e02-b966-aa28d31d36dc-kube-api-access-7vp9s" (OuterVolumeSpecName: "kube-api-access-7vp9s") pod "9c886d69-8bb4-4e02-b966-aa28d31d36dc" (UID: "9c886d69-8bb4-4e02-b966-aa28d31d36dc"). InnerVolumeSpecName "kube-api-access-7vp9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:37:23.625757 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.625728    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:37:23.694497 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19efbd9-82bf-4a27-8f9a-c542633bbac3-plugins-config-volume\") pod \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\" (UID: \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\") "
Jul 24 23:37:23.694726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/144d2566-8c25-419b-930f-c64fa9def932-openshift-service-ca\") pod \"144d2566-8c25-419b-930f-c64fa9def932\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") "
Jul 24 23:37:23.694726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694579    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svlnz\" (UniqueName: \"kubernetes.io/projected/144d2566-8c25-419b-930f-c64fa9def932-kube-api-access-svlnz\") pod \"144d2566-8c25-419b-930f-c64fa9def932\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") "
Jul 24 23:37:23.694726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144d2566-8c25-419b-930f-c64fa9def932-maas-api-tls\") pod \"144d2566-8c25-419b-930f-c64fa9def932\" (UID: \"144d2566-8c25-419b-930f-c64fa9def932\") "
Jul 24 23:37:23.694726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694681    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km22t\" (UniqueName: \"kubernetes.io/projected/d19efbd9-82bf-4a27-8f9a-c542633bbac3-kube-api-access-km22t\") pod \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\" (UID: \"d19efbd9-82bf-4a27-8f9a-c542633bbac3\") "
Jul 24 23:37:23.694937 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694855    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19efbd9-82bf-4a27-8f9a-c542633bbac3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d19efbd9-82bf-4a27-8f9a-c542633bbac3" (UID: "d19efbd9-82bf-4a27-8f9a-c542633bbac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:23.694937 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144d2566-8c25-419b-930f-c64fa9def932-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "144d2566-8c25-419b-930f-c64fa9def932" (UID: "144d2566-8c25-419b-930f-c64fa9def932"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:23.694937 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694914    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vp9s\" (UniqueName: \"kubernetes.io/projected/9c886d69-8bb4-4e02-b966-aa28d31d36dc-kube-api-access-7vp9s\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:23.695045 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.694955    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c886d69-8bb4-4e02-b966-aa28d31d36dc-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:23.696852 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.696823    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144d2566-8c25-419b-930f-c64fa9def932-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "144d2566-8c25-419b-930f-c64fa9def932" (UID: "144d2566-8c25-419b-930f-c64fa9def932"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:37:23.696943 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.696920    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19efbd9-82bf-4a27-8f9a-c542633bbac3-kube-api-access-km22t" (OuterVolumeSpecName: "kube-api-access-km22t") pod "d19efbd9-82bf-4a27-8f9a-c542633bbac3" (UID: "d19efbd9-82bf-4a27-8f9a-c542633bbac3"). InnerVolumeSpecName "kube-api-access-km22t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:37:23.697090 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.697074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144d2566-8c25-419b-930f-c64fa9def932-kube-api-access-svlnz" (OuterVolumeSpecName: "kube-api-access-svlnz") pod "144d2566-8c25-419b-930f-c64fa9def932" (UID: "144d2566-8c25-419b-930f-c64fa9def932"). InnerVolumeSpecName "kube-api-access-svlnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:37:23.795879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.795770    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svlnz\" (UniqueName: \"kubernetes.io/projected/144d2566-8c25-419b-930f-c64fa9def932-kube-api-access-svlnz\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:23.795879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.795805    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144d2566-8c25-419b-930f-c64fa9def932-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:23.795879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.795818    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km22t\" (UniqueName: \"kubernetes.io/projected/d19efbd9-82bf-4a27-8f9a-c542633bbac3-kube-api-access-km22t\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:23.795879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.795829    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19efbd9-82bf-4a27-8f9a-c542633bbac3-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:23.795879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.795841    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/144d2566-8c25-419b-930f-c64fa9def932-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:23.844183 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:23.844149    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aaa525f-cbb1-47db-81fd-f06e1bd067b0" path="/var/lib/kubelet/pods/3aaa525f-cbb1-47db-81fd-f06e1bd067b0/volumes"
Jul 24 23:37:24.326962 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.326934    2563 scope.go:117] "RemoveContainer" containerID="a037817cae8d8678feea24f87ac9171ecd57ebb12170362a478d62f16d3a0702"
Jul 24 23:37:24.335276 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.335256    2563 scope.go:117] "RemoveContainer" containerID="9f643234b81ebcafdca5f44076f5f88ae146e55ce811a681367f28646edeadfb"
Jul 24 23:37:24.342772 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.342754    2563 scope.go:117] "RemoveContainer" containerID="5d86b7634e29deab0fef7c7b64b538a3ab7a29b7e139c17fa4a8769c4d5a9b9f"
Jul 24 23:37:24.350243 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.350227    2563 scope.go:117] "RemoveContainer" containerID="4cfffe3d8d64d5edcd7695217d0f96385a75487066e4aa55bbcf437c6082986b"
Jul 24 23:37:24.492540 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.492493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j" event={"ID":"d19efbd9-82bf-4a27-8f9a-c542633bbac3","Type":"ContainerDied","Data":"30249148d430b1829b9da1c26ed5cc3ab98ff258f22504e09df873482b474cdb"}
Jul 24 23:37:24.492721 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.492574    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs" event={"ID":"144d2566-8c25-419b-930f-c64fa9def932","Type":"ContainerDied","Data":"7e1e5320c462553bc568c7a262359694b2b8b50ef3419b7917c087810d64e5f7"}
Jul 24 23:37:24.492721 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.492584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"
Jul 24 23:37:24.492721 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.492598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c" event={"ID":"9c886d69-8bb4-4e02-b966-aa28d31d36dc","Type":"ContainerDied","Data":"f16bb83f11f18601552e23d7ae94c639273acaae93dc9f97d366aa4528d723dd"}
Jul 24 23:37:24.492721 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.492572    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"
Jul 24 23:37:24.492721 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.492669    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"
Jul 24 23:37:24.535619 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.532544    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"]
Jul 24 23:37:24.538046 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.538019    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-5c850d-57d95759b-7f2xs"]
Jul 24 23:37:24.559414 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.559381    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"]
Jul 24 23:37:24.561804 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.561781    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-5c850d-87bc5969b-fhn7j"]
Jul 24 23:37:24.571335 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.571314    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"]
Jul 24 23:37:24.577011 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:24.576988    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-5c850d-87c54d96b-4rm6c"]
Jul 24 23:37:25.842585 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:25.842551    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144d2566-8c25-419b-930f-c64fa9def932" path="/var/lib/kubelet/pods/144d2566-8c25-419b-930f-c64fa9def932/volumes"
Jul 24 23:37:25.842961 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:25.842894    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c886d69-8bb4-4e02-b966-aa28d31d36dc" path="/var/lib/kubelet/pods/9c886d69-8bb4-4e02-b966-aa28d31d36dc/volumes"
Jul 24 23:37:25.843186 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:25.843173    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19efbd9-82bf-4a27-8f9a-c542633bbac3" path="/var/lib/kubelet/pods/d19efbd9-82bf-4a27-8f9a-c542633bbac3/volumes"
Jul 24 23:37:26.421025 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:26.420991    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:37:27.939021 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:27.938986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:28.314647 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:28.314561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:33.969201 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:33.969165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:34.029629 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.029598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:34.229145 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.229060    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:34.453822 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.453794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:34.582328 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582238    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"]
Jul 24 23:37:34.582655 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582641    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c886d69-8bb4-4e02-b966-aa28d31d36dc" containerName="payload-processing"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582659    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c886d69-8bb4-4e02-b966-aa28d31d36dc" containerName="payload-processing"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582669    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="254fac1f-8d2a-48b4-acfd-96a41c354564" containerName="maas-api"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582675    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="254fac1f-8d2a-48b4-acfd-96a41c354564" containerName="maas-api"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582683    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61dc299e-1570-4b8d-845a-0a6d67082439" containerName="revoke-keys"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582689    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61dc299e-1570-4b8d-845a-0a6d67082439" containerName="revoke-keys"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582698    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144d2566-8c25-419b-930f-c64fa9def932" containerName="maas-api"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582703    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="144d2566-8c25-419b-930f-c64fa9def932" containerName="maas-api"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582721    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9303319e-47eb-404e-af84-5d0a2ac92a71" containerName="payload-pre-processing"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582727    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9303319e-47eb-404e-af84-5d0a2ac92a71" containerName="payload-pre-processing"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582738    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19efbd9-82bf-4a27-8f9a-c542633bbac3" containerName="payload-pre-processing"
Jul 24 23:37:34.582737 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582743    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19efbd9-82bf-4a27-8f9a-c542633bbac3" containerName="payload-pre-processing"
Jul 24 23:37:34.583147 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582806    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="254fac1f-8d2a-48b4-acfd-96a41c354564" containerName="maas-api"
Jul 24 23:37:34.583147 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582815    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="144d2566-8c25-419b-930f-c64fa9def932" containerName="maas-api"
Jul 24 23:37:34.583147 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582820    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c886d69-8bb4-4e02-b966-aa28d31d36dc" containerName="payload-processing"
Jul 24 23:37:34.583147 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582826    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9303319e-47eb-404e-af84-5d0a2ac92a71" containerName="payload-pre-processing"
Jul 24 23:37:34.583147 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582832    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d19efbd9-82bf-4a27-8f9a-c542633bbac3" containerName="payload-pre-processing"
Jul 24 23:37:34.583147 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.582838    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61dc299e-1570-4b8d-845a-0a6d67082439" containerName="revoke-keys"
Jul 24 23:37:34.585844 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.585825    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"
Jul 24 23:37:34.588367 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.588343    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\""
Jul 24 23:37:34.592804 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.592763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"]
Jul 24 23:37:34.689829 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.689794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40219daa-88c6-4c5f-a78d-2d2ab8822c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2\" (UID: \"40219daa-88c6-4c5f-a78d-2d2ab8822c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"
Jul 24 23:37:34.790928 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.790887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40219daa-88c6-4c5f-a78d-2d2ab8822c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2\" (UID: \"40219daa-88c6-4c5f-a78d-2d2ab8822c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"
Jul 24 23:37:34.791620 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.791598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40219daa-88c6-4c5f-a78d-2d2ab8822c62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2\" (UID: \"40219daa-88c6-4c5f-a78d-2d2ab8822c62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"
Jul 24 23:37:34.900649 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:34.900614    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"
Jul 24 23:37:35.038810 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:35.038786    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"]
Jul 24 23:37:35.040794 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:37:35.040750    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40219daa_88c6_4c5f_a78d_2d2ab8822c62.slice/crio-ed7971c4900b22276ee73e7dd4cd51d731688d49226af9475d445f9b49d256fd WatchSource:0}: Error finding container ed7971c4900b22276ee73e7dd4cd51d731688d49226af9475d445f9b49d256fd: Status 404 returned error can't find the container with id ed7971c4900b22276ee73e7dd4cd51d731688d49226af9475d445f9b49d256fd
Jul 24 23:37:35.548598 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:35.548487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2" event={"ID":"40219daa-88c6-4c5f-a78d-2d2ab8822c62","Type":"ContainerStarted","Data":"a41b5313f168e92c487a00f6439420b8b25f8956e0974786c76750639483cfc0"}
Jul 24 23:37:35.548598 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:35.548564    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2" event={"ID":"40219daa-88c6-4c5f-a78d-2d2ab8822c62","Type":"ContainerStarted","Data":"ed7971c4900b22276ee73e7dd4cd51d731688d49226af9475d445f9b49d256fd"}
Jul 24 23:37:35.563866 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:35.563820    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2" podStartSLOduration=1.563804041 podStartE2EDuration="1.563804041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:37:35.563020892 +0000 UTC m=+3072.196299844" watchObservedRunningTime="2026-07-24 23:37:35.563804041 +0000 UTC m=+3072.197082981"
Jul 24 23:37:36.555150 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:36.555119    2563 generic.go:356] "Generic (PLEG): container finished" podID="40219daa-88c6-4c5f-a78d-2d2ab8822c62" containerID="a41b5313f168e92c487a00f6439420b8b25f8956e0974786c76750639483cfc0" exitCode=0
Jul 24 23:37:36.555581 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:36.555167    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2" event={"ID":"40219daa-88c6-4c5f-a78d-2d2ab8822c62","Type":"ContainerDied","Data":"a41b5313f168e92c487a00f6439420b8b25f8956e0974786c76750639483cfc0"}
Jul 24 23:37:37.681002 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:37.680978    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"
Jul 24 23:37:37.819251 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:37.819153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40219daa-88c6-4c5f-a78d-2d2ab8822c62-maas-api-service-ca\") pod \"40219daa-88c6-4c5f-a78d-2d2ab8822c62\" (UID: \"40219daa-88c6-4c5f-a78d-2d2ab8822c62\") "
Jul 24 23:37:37.819573 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:37.819494    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40219daa-88c6-4c5f-a78d-2d2ab8822c62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40219daa-88c6-4c5f-a78d-2d2ab8822c62" (UID: "40219daa-88c6-4c5f-a78d-2d2ab8822c62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:37.920893 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:37.920850    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40219daa-88c6-4c5f-a78d-2d2ab8822c62-maas-api-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:38.566357 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:38.566320    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"
Jul 24 23:37:38.566357 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:38.566344    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2" event={"ID":"40219daa-88c6-4c5f-a78d-2d2ab8822c62","Type":"ContainerDied","Data":"ed7971c4900b22276ee73e7dd4cd51d731688d49226af9475d445f9b49d256fd"}
Jul 24 23:37:38.566580 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:38.566388    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed7971c4900b22276ee73e7dd4cd51d731688d49226af9475d445f9b49d256fd"
Jul 24 23:37:40.997908 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:40.997815    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:41.099082 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:41.099041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:45.095911 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.095875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"]
Jul 24 23:37:45.098585 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.096613    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9" podUID="9f2fb6f1-350c-438b-892e-173ae07d2965" containerName="maas-api" containerID="cri-o://5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987" gracePeriod=30
Jul 24 23:37:45.126608 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.126574    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"]
Jul 24 23:37:45.128765 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.128720    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d08a26-29748930-pxx56"]
Jul 24 23:37:45.132299 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.132273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:45.160807 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.160770    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"]
Jul 24 23:37:45.160989 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.160970    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp" podUID="1d5ac741-7edc-4b79-92b5-ed7b300de390" containerName="payload-processing" containerID="cri-o://b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46" gracePeriod=30
Jul 24 23:37:45.166939 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.166906    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"]
Jul 24 23:37:45.167157 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.167131    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp" podUID="b060cdea-c5cb-4bf3-b91e-edd36b9673c7" containerName="payload-pre-processing" containerID="cri-o://c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3" gracePeriod=30
Jul 24 23:37:45.201089 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.201055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:45.260042 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.260013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:45.434683 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.434656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:37:45.455930 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.455899    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:37:45.466069 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.466044    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:37:45.587145 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2fb6f1-350c-438b-892e-173ae07d2965-openshift-service-ca\") pod \"9f2fb6f1-350c-438b-892e-173ae07d2965\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") "
Jul 24 23:37:45.587345 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587160    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stmll\" (UniqueName: \"kubernetes.io/projected/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-kube-api-access-stmll\") pod \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\" (UID: \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\") "
Jul 24 23:37:45.587345 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rzvj\" (UniqueName: \"kubernetes.io/projected/9f2fb6f1-350c-438b-892e-173ae07d2965-kube-api-access-4rzvj\") pod \"9f2fb6f1-350c-438b-892e-173ae07d2965\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") "
Jul 24 23:37:45.587345 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587252    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-plugins-config-volume\") pod \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\" (UID: \"b060cdea-c5cb-4bf3-b91e-edd36b9673c7\") "
Jul 24 23:37:45.587345 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587288    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5ac741-7edc-4b79-92b5-ed7b300de390-plugins-config-volume\") pod \"1d5ac741-7edc-4b79-92b5-ed7b300de390\" (UID: \"1d5ac741-7edc-4b79-92b5-ed7b300de390\") "
Jul 24 23:37:45.587345 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587334    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7wzr\" (UniqueName: \"kubernetes.io/projected/1d5ac741-7edc-4b79-92b5-ed7b300de390-kube-api-access-v7wzr\") pod \"1d5ac741-7edc-4b79-92b5-ed7b300de390\" (UID: \"1d5ac741-7edc-4b79-92b5-ed7b300de390\") "
Jul 24 23:37:45.587701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls\") pod \"9f2fb6f1-350c-438b-892e-173ae07d2965\" (UID: \"9f2fb6f1-350c-438b-892e-173ae07d2965\") "
Jul 24 23:37:45.587701 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2fb6f1-350c-438b-892e-173ae07d2965-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f2fb6f1-350c-438b-892e-173ae07d2965" (UID: "9f2fb6f1-350c-438b-892e-173ae07d2965"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:45.587813 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b060cdea-c5cb-4bf3-b91e-edd36b9673c7" (UID: "b060cdea-c5cb-4bf3-b91e-edd36b9673c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:45.587813 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.587761    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d5ac741-7edc-4b79-92b5-ed7b300de390-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d5ac741-7edc-4b79-92b5-ed7b300de390" (UID: "1d5ac741-7edc-4b79-92b5-ed7b300de390"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 23:37:45.589488 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.589458    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d5ac741-7edc-4b79-92b5-ed7b300de390-kube-api-access-v7wzr" (OuterVolumeSpecName: "kube-api-access-v7wzr") pod "1d5ac741-7edc-4b79-92b5-ed7b300de390" (UID: "1d5ac741-7edc-4b79-92b5-ed7b300de390"). InnerVolumeSpecName "kube-api-access-v7wzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:37:45.589488 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.589482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-kube-api-access-stmll" (OuterVolumeSpecName: "kube-api-access-stmll") pod "b060cdea-c5cb-4bf3-b91e-edd36b9673c7" (UID: "b060cdea-c5cb-4bf3-b91e-edd36b9673c7"). InnerVolumeSpecName "kube-api-access-stmll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:37:45.589643 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.589501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2fb6f1-350c-438b-892e-173ae07d2965-kube-api-access-4rzvj" (OuterVolumeSpecName: "kube-api-access-4rzvj") pod "9f2fb6f1-350c-438b-892e-173ae07d2965" (UID: "9f2fb6f1-350c-438b-892e-173ae07d2965"). InnerVolumeSpecName "kube-api-access-4rzvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 23:37:45.589643 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.589507    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f2fb6f1-350c-438b-892e-173ae07d2965" (UID: "9f2fb6f1-350c-438b-892e-173ae07d2965"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 23:37:45.606929 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.606903    2563 generic.go:356] "Generic (PLEG): container finished" podID="b060cdea-c5cb-4bf3-b91e-edd36b9673c7" containerID="c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3" exitCode=0
Jul 24 23:37:45.607088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.606950    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"
Jul 24 23:37:45.607088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.606975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp" event={"ID":"b060cdea-c5cb-4bf3-b91e-edd36b9673c7","Type":"ContainerDied","Data":"c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3"}
Jul 24 23:37:45.607088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.607055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp" event={"ID":"b060cdea-c5cb-4bf3-b91e-edd36b9673c7","Type":"ContainerDied","Data":"88d2dc358145e631b9c717671ae11241486c25e7e419366093c2aa568e983edd"}
Jul 24 23:37:45.607088 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.607080    2563 scope.go:117] "RemoveContainer" containerID="c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3"
Jul 24 23:37:45.608248 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.608228    2563 generic.go:356] "Generic (PLEG): container finished" podID="9f2fb6f1-350c-438b-892e-173ae07d2965" containerID="5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987" exitCode=0
Jul 24 23:37:45.608359 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.608261    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9" event={"ID":"9f2fb6f1-350c-438b-892e-173ae07d2965","Type":"ContainerDied","Data":"5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987"}
Jul 24 23:37:45.608359 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.608283    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"
Jul 24 23:37:45.608359 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.608310    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9" event={"ID":"9f2fb6f1-350c-438b-892e-173ae07d2965","Type":"ContainerDied","Data":"0f68942fedf0ee81443817fd89a45ed915f50b2484c3ff09c1abfcb65c543cbc"}
Jul 24 23:37:45.609864 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.609840    2563 generic.go:356] "Generic (PLEG): container finished" podID="1d5ac741-7edc-4b79-92b5-ed7b300de390" containerID="b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46" exitCode=0
Jul 24 23:37:45.609980 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.609881    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"
Jul 24 23:37:45.609980 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.609878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp" event={"ID":"1d5ac741-7edc-4b79-92b5-ed7b300de390","Type":"ContainerDied","Data":"b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46"}
Jul 24 23:37:45.610100 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.609988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp" event={"ID":"1d5ac741-7edc-4b79-92b5-ed7b300de390","Type":"ContainerDied","Data":"edef63e6b6d0eb1a7df53e9d16f3497f442b80df432a8f09424310f57c562540"}
Jul 24 23:37:45.618980 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.618961    2563 scope.go:117] "RemoveContainer" containerID="c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3"
Jul 24 23:37:45.619412 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:37:45.619387    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3\": container with ID starting with c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3 not found: ID does not exist" containerID="c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3"
Jul 24 23:37:45.619499 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.619423    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3"} err="failed to get container status \"c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3\": rpc error: code = NotFound desc = could not find container \"c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3\": container with ID starting with c2bb3b66a3852bc8bd6ec5e9423a6cb21729d498985b52c49c0dbba4d54708b3 not found: ID does not exist"
Jul 24 23:37:45.619499 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.619456    2563 scope.go:117] "RemoveContainer" containerID="5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987"
Jul 24 23:37:45.628494 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.628475    2563 scope.go:117] "RemoveContainer" containerID="5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987"
Jul 24 23:37:45.628787 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:37:45.628769    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987\": container with ID starting with 5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987 not found: ID does not exist" containerID="5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987"
Jul 24 23:37:45.628861 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.628807    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987"} err="failed to get container status \"5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987\": rpc error: code = NotFound desc = could not find container \"5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987\": container with ID starting with 5a4f761e00a622da3c8cf1f618c142bd84adf5af0a7066761d5cb7b03595c987 not found: ID does not exist"
Jul 24 23:37:45.628861 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.628823    2563 scope.go:117] "RemoveContainer" containerID="b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46"
Jul 24 23:37:45.636079 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.636060    2563 scope.go:117] "RemoveContainer" containerID="b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46"
Jul 24 23:37:45.636423 ip-10-0-136-221 kubenswrapper[2563]: E0724 23:37:45.636387    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46\": container with ID starting with b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46 not found: ID does not exist" containerID="b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46"
Jul 24 23:37:45.636590 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.636420    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46"} err="failed to get container status \"b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46\": rpc error: code = NotFound desc = could not find container \"b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46\": container with ID starting with b7caae17808de302f69b5b9c5878f12d2c7f5ec62dd302b5e912f70eb8e72b46 not found: ID does not exist"
Jul 24 23:37:45.638194 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.638174    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"]
Jul 24 23:37:45.642152 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.642131    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d08a26-5d4f8667-nzsk9"]
Jul 24 23:37:45.656533 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.656494    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"]
Jul 24 23:37:45.659629 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.659606    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-d08a26-6dc999d9b-ptltp"]
Jul 24 23:37:45.669726 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.669699    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"]
Jul 24 23:37:45.672805 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.672784    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-d08a26-579fdb5678-vdwlp"]
Jul 24 23:37:45.688803 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.688744    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:45.688803 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.688780    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d5ac741-7edc-4b79-92b5-ed7b300de390-plugins-config-volume\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:45.688803 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.688810    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wzr\" (UniqueName: \"kubernetes.io/projected/1d5ac741-7edc-4b79-92b5-ed7b300de390-kube-api-access-v7wzr\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:45.689017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.688824    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f2fb6f1-350c-438b-892e-173ae07d2965-maas-api-tls\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:45.689017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.688838    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f2fb6f1-350c-438b-892e-173ae07d2965-openshift-service-ca\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:45.689017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.688850    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stmll\" (UniqueName: \"kubernetes.io/projected/b060cdea-c5cb-4bf3-b91e-edd36b9673c7-kube-api-access-stmll\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:45.689017 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.688862    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rzvj\" (UniqueName: \"kubernetes.io/projected/9f2fb6f1-350c-438b-892e-173ae07d2965-kube-api-access-4rzvj\") on node \"ip-10-0-136-221.ec2.internal\" DevicePath \"\""
Jul 24 23:37:45.841146 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.841105    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d5ac741-7edc-4b79-92b5-ed7b300de390" path="/var/lib/kubelet/pods/1d5ac741-7edc-4b79-92b5-ed7b300de390/volumes"
Jul 24 23:37:45.841464 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.841446    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908a0de9-a26a-4080-8d17-78608a9412ea" path="/var/lib/kubelet/pods/908a0de9-a26a-4080-8d17-78608a9412ea/volumes"
Jul 24 23:37:45.841826 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.841808    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2fb6f1-350c-438b-892e-173ae07d2965" path="/var/lib/kubelet/pods/9f2fb6f1-350c-438b-892e-173ae07d2965/volumes"
Jul 24 23:37:45.842161 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:45.842145    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b060cdea-c5cb-4bf3-b91e-edd36b9673c7" path="/var/lib/kubelet/pods/b060cdea-c5cb-4bf3-b91e-edd36b9673c7/volumes"
Jul 24 23:37:47.930285 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:47.930251    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-dmgs6"]
Jul 24 23:37:49.648639 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:49.648610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:49.653738 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:49.653713    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"]
Jul 24 23:37:49.655805 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:49.655782    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d08a26-rjcr2"]
Jul 24 23:37:49.841353 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:49.841320    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40219daa-88c6-4c5f-a78d-2d2ab8822c62" path="/var/lib/kubelet/pods/40219daa-88c6-4c5f-a78d-2d2ab8822c62/volumes"
Jul 24 23:37:50.053812 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:50.053733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:55.727680 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:55.727645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:55.790792 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:55.790763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:55.993264 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:55.993182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:37:56.216118 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:37:56.216084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:38:01.893856 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:01.893826    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-g2bfn_12599807-0bc5-4b17-b082-c1d241824eda/manager/0.log"
Jul 24 23:38:02.006827 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:02.006797    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-24r2g_b92091da-84f6-47e5-b995-0d9090585a9f/manager/0.log"
Jul 24 23:38:02.737266 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:02.737236    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d99768bd-l4fxb_2f5f046f-7d2a-457e-a2f4-34b2bacdd803/copy-manifests/0.log"
Jul 24 23:38:02.751712 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:02.751682    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d99768bd-l4fxb_2f5f046f-7d2a-457e-a2f4-34b2bacdd803/manager/0.log"
Jul 24 23:38:03.208979 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:03.208950    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748900-7nm6k_397f99d4-68ac-40a5-a639-dd36ed36ac4a/cleanup/0.log"
Jul 24 23:38:03.318602 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:03.318560    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748915-jx545_2d640af8-5143-4b0d-87fc-9bc1657fca00/cleanup/0.log"
Jul 24 23:38:03.426529 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:03.426500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748930-9dgz5_6f494848-e793-4b97-9993-f1417d2a2066/cleanup/0.log"
Jul 24 23:38:05.496681 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:05.496649    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dmgs6_af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35/limitador/0.log"
Jul 24 23:38:05.615855 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:05.615821    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kzwnr_3752acfb-9605-4c60-8f05-262b01003338/manager/0.log"
Jul 24 23:38:06.095980 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:06.095948    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p5556_1aef2698-81d6-4bd8-8fbb-43ca5b78af17/discovery/0.log"
Jul 24 23:38:06.319886 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:06.319855    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849489fbdc-qw8vb_4e0160b2-3818-4707-95a8-75946cc4f81f/kube-auth-proxy/0.log"
Jul 24 23:38:07.386745 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:07.386706    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tkdz4_28ef1232-1949-4b38-96d4-0a38ceb9b805/main/0.log"
Jul 24 23:38:13.301876 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:13.301842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:38:13.365740 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:13.365706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log" limit=1024
Jul 24 23:38:16.154355 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:16.154322    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rv5zf_303ae2ca-2b38-4094-9507-21820dc52eeb/global-pull-secret-syncer/0.log"
Jul 24 23:38:16.244382 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:16.244347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-85c4k_31ca8850-9999-4ca9-be05-4dc4f733d845/konnectivity-agent/0.log"
Jul 24 23:38:16.390358 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:16.390324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-221.ec2.internal_5b24225b2a0a08b20d14ab2b1793c0b0/haproxy/0.log"
Jul 24 23:38:22.357929 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:22.357901    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-dmgs6_af4d8ff7-ffe2-4a2d-aee2-ad0ed59fdc35/limitador/0.log"
Jul 24 23:38:22.429573 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:22.429513    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kzwnr_3752acfb-9605-4c60-8f05-262b01003338/manager/0.log"
Jul 24 23:38:24.303073 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.303044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-pkd6d_ceb3f17a-e8eb-4013-99c6-bf2ac5a82baf/monitoring-plugin/0.log"
Jul 24 23:38:24.386185 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.386156    2563 scope.go:117] "RemoveContainer" containerID="ab28fb3d54ee6803f17e0f9041dae22be547ddffcd34e71e443ae796d608385d"
Jul 24 23:38:24.399706 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.399685    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q22z2_9570cf0b-24f4-4b27-a89a-018d38a45a1d/node-exporter/0.log"
Jul 24 23:38:24.426046 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.426015    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q22z2_9570cf0b-24f4-4b27-a89a-018d38a45a1d/kube-rbac-proxy/0.log"
Jul 24 23:38:24.447262 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.447235    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q22z2_9570cf0b-24f4-4b27-a89a-018d38a45a1d/init-textfile/0.log"
Jul 24 23:38:24.929940 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.929911    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7848bb556d-skdzx_2f37c0e4-fe28-4382-9e8b-81667a365887/thanos-query/0.log"
Jul 24 23:38:24.950480 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.950451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7848bb556d-skdzx_2f37c0e4-fe28-4382-9e8b-81667a365887/kube-rbac-proxy-web/0.log"
Jul 24 23:38:24.970593 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.970563    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7848bb556d-skdzx_2f37c0e4-fe28-4382-9e8b-81667a365887/kube-rbac-proxy/0.log"
Jul 24 23:38:24.991435 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:24.991411    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7848bb556d-skdzx_2f37c0e4-fe28-4382-9e8b-81667a365887/prom-label-proxy/0.log"
Jul 24 23:38:25.012988 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:25.012964    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7848bb556d-skdzx_2f37c0e4-fe28-4382-9e8b-81667a365887/kube-rbac-proxy-rules/0.log"
Jul 24 23:38:25.034797 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:25.034767    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7848bb556d-skdzx_2f37c0e4-fe28-4382-9e8b-81667a365887/kube-rbac-proxy-metrics/0.log"
Jul 24 23:38:26.007861 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.007828    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"]
Jul 24 23:38:26.008290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008198    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2fb6f1-350c-438b-892e-173ae07d2965" containerName="maas-api"
Jul 24 23:38:26.008290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008214    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2fb6f1-350c-438b-892e-173ae07d2965" containerName="maas-api"
Jul 24 23:38:26.008290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008227    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b060cdea-c5cb-4bf3-b91e-edd36b9673c7" containerName="payload-pre-processing"
Jul 24 23:38:26.008290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008233    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b060cdea-c5cb-4bf3-b91e-edd36b9673c7" containerName="payload-pre-processing"
Jul 24 23:38:26.008290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008240    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d5ac741-7edc-4b79-92b5-ed7b300de390" containerName="payload-processing"
Jul 24 23:38:26.008290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008246    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d5ac741-7edc-4b79-92b5-ed7b300de390" containerName="payload-processing"
Jul 24 23:38:26.008290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008252    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40219daa-88c6-4c5f-a78d-2d2ab8822c62" containerName="revoke-keys"
Jul 24 23:38:26.008290 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008257    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="40219daa-88c6-4c5f-a78d-2d2ab8822c62" containerName="revoke-keys"
Jul 24 23:38:26.008583 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008313    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2fb6f1-350c-438b-892e-173ae07d2965" containerName="maas-api"
Jul 24 23:38:26.008583 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008323    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d5ac741-7edc-4b79-92b5-ed7b300de390" containerName="payload-processing"
Jul 24 23:38:26.008583 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008331    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b060cdea-c5cb-4bf3-b91e-edd36b9673c7" containerName="payload-pre-processing"
Jul 24 23:38:26.008583 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.008339    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="40219daa-88c6-4c5f-a78d-2d2ab8822c62" containerName="revoke-keys"
Jul 24 23:38:26.013257 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.013228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.016129 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.016104    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qntww\"/\"openshift-service-ca.crt\""
Jul 24 23:38:26.016305 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.016160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qntww\"/\"default-dockercfg-cs6ws\""
Jul 24 23:38:26.017356 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.017336    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qntww\"/\"kube-root-ca.crt\""
Jul 24 23:38:26.021128 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.021107    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"]
Jul 24 23:38:26.126682 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.126624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-sys\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.126682 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.126688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-lib-modules\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.126915 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.126777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-podres\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.126915 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.126811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-proc\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.126915 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.126846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjzmm\" (UniqueName: \"kubernetes.io/projected/da1970e8-01f8-41d3-9c9d-9c938c82c505-kube-api-access-gjzmm\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.227690 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjzmm\" (UniqueName: \"kubernetes.io/projected/da1970e8-01f8-41d3-9c9d-9c938c82c505-kube-api-access-gjzmm\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.227856 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-sys\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.227856 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-lib-modules\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.227856 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-podres\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.227856 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227796    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-proc\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.227856 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-sys\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.228032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227877    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-proc\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.228032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-lib-modules\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.228032 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.227925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da1970e8-01f8-41d3-9c9d-9c938c82c505-podres\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.235537 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.235487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjzmm\" (UniqueName: \"kubernetes.io/projected/da1970e8-01f8-41d3-9c9d-9c938c82c505-kube-api-access-gjzmm\") pod \"perf-node-gather-daemonset-zzc5f\" (UID: \"da1970e8-01f8-41d3-9c9d-9c938c82c505\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.323412 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.323319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.457472 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.457445    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"]
Jul 24 23:38:26.460166 ip-10-0-136-221 kubenswrapper[2563]: W0724 23:38:26.460140    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda1970e8_01f8_41d3_9c9d_9c938c82c505.slice/crio-7e48aa6059aba92a5291a1f9b2afb1f8e72df372e5dc023c5e133993dfaedc0d WatchSource:0}: Error finding container 7e48aa6059aba92a5291a1f9b2afb1f8e72df372e5dc023c5e133993dfaedc0d: Status 404 returned error can't find the container with id 7e48aa6059aba92a5291a1f9b2afb1f8e72df372e5dc023c5e133993dfaedc0d
Jul 24 23:38:26.811369 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.811332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f" event={"ID":"da1970e8-01f8-41d3-9c9d-9c938c82c505","Type":"ContainerStarted","Data":"c6ad702f6b764eea677b18aa220bb59cd419e70508fde3f32a12389485be7d2f"}
Jul 24 23:38:26.811620 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.811390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f" event={"ID":"da1970e8-01f8-41d3-9c9d-9c938c82c505","Type":"ContainerStarted","Data":"7e48aa6059aba92a5291a1f9b2afb1f8e72df372e5dc023c5e133993dfaedc0d"}
Jul 24 23:38:26.811620 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.811427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:26.830659 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:26.830607    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f" podStartSLOduration=1.830589695 podStartE2EDuration="1.830589695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:38:26.828954905 +0000 UTC m=+3123.462233844" watchObservedRunningTime="2026-07-24 23:38:26.830589695 +0000 UTC m=+3123.463868645"
Jul 24 23:38:27.321360 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:27.321324    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67547fd788-5g5mm_426e6fa9-09a2-499f-9e7d-d00a18e0217e/console/0.log"
Jul 24 23:38:28.906560 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:28.906512    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-blcb4_2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca/dns/0.log"
Jul 24 23:38:28.927279 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:28.927251    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-blcb4_2b6a49d4-1ac6-49d1-b540-bf2ad1d7b8ca/kube-rbac-proxy/0.log"
Jul 24 23:38:28.998569 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:28.998515    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6lphx_06426582-6029-4f0b-a88f-eea7b0a829d6/dns-node-resolver/0.log"
Jul 24 23:38:29.657398 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:29.657368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8qv9m_1746104c-a257-4d75-ba28-02305cbb01ba/node-ca/0.log"
Jul 24 23:38:30.718182 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:30.718145    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p5556_1aef2698-81d6-4bd8-8fbb-43ca5b78af17/discovery/0.log"
Jul 24 23:38:30.764124 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:30.764092    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849489fbdc-qw8vb_4e0160b2-3818-4707-95a8-75946cc4f81f/kube-auth-proxy/0.log"
Jul 24 23:38:31.749741 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:31.749712    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r5g4j_fbe5e964-56ec-4897-8b14-7d09a515919e/serve-healthcheck-canary/0.log"
Jul 24 23:38:32.304283 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:32.304256    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jzzs7_310b20a4-9c4c-44ec-bb15-f2c86b76e64a/kube-rbac-proxy/0.log"
Jul 24 23:38:32.329992 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:32.329965    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jzzs7_310b20a4-9c4c-44ec-bb15-f2c86b76e64a/exporter/0.log"
Jul 24 23:38:32.351224 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:32.351198    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jzzs7_310b20a4-9c4c-44ec-bb15-f2c86b76e64a/extractor/0.log"
Jul 24 23:38:32.827449 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:32.827415    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-zzc5f"
Jul 24 23:38:34.614898 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:34.614873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-g2bfn_12599807-0bc5-4b17-b082-c1d241824eda/manager/0.log"
Jul 24 23:38:34.828985 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:34.828952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-24r2g_b92091da-84f6-47e5-b995-0d9090585a9f/manager/0.log"
Jul 24 23:38:35.362689 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:35.362656    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d99768bd-l4fxb_2f5f046f-7d2a-457e-a2f4-34b2bacdd803/manager/0.log"
Jul 24 23:38:35.382531 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:35.382491    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d99768bd-l4fxb_2f5f046f-7d2a-457e-a2f4-34b2bacdd803/copy-manifests/0.log"
Jul 24 23:38:36.526202 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:36.526172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p5556_1aef2698-81d6-4bd8-8fbb-43ca5b78af17/discovery/0.log"
Jul 24 23:38:36.549901 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:36.549875    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849489fbdc-qw8vb_4e0160b2-3818-4707-95a8-75946cc4f81f/kube-auth-proxy/0.log"
Jul 24 23:38:38.220376 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:38.220341    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p5556_1aef2698-81d6-4bd8-8fbb-43ca5b78af17/discovery/0.log"
Jul 24 23:38:38.249085 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:38.249062    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849489fbdc-qw8vb_4e0160b2-3818-4707-95a8-75946cc4f81f/kube-auth-proxy/0.log"
Jul 24 23:38:42.961509 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:42.961479    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2gwg8_f8634fe9-3c65-480b-ab1c-ab306e77b9f6/kube-multus/0.log"
Jul 24 23:38:43.020620 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.020590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bf8x_cd192207-7476-4816-a50d-ec3a230d5d73/kube-multus-additional-cni-plugins/0.log"
Jul 24 23:38:43.041873 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.041845    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bf8x_cd192207-7476-4816-a50d-ec3a230d5d73/egress-router-binary-copy/0.log"
Jul 24 23:38:43.061879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.061851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bf8x_cd192207-7476-4816-a50d-ec3a230d5d73/cni-plugins/0.log"
Jul 24 23:38:43.082687 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.082659    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bf8x_cd192207-7476-4816-a50d-ec3a230d5d73/bond-cni-plugin/0.log"
Jul 24 23:38:43.103948 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.103921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bf8x_cd192207-7476-4816-a50d-ec3a230d5d73/routeoverride-cni/0.log"
Jul 24 23:38:43.124879 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.124851    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bf8x_cd192207-7476-4816-a50d-ec3a230d5d73/whereabouts-cni-bincopy/0.log"
Jul 24 23:38:43.145551 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.145511    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5bf8x_cd192207-7476-4816-a50d-ec3a230d5d73/whereabouts-cni/0.log"
Jul 24 23:38:43.563052 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.563025    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nspfm_35f71564-7bff-4960-a63b-584726e054eb/network-metrics-daemon/0.log"
Jul 24 23:38:43.585415 ip-10-0-136-221 kubenswrapper[2563]: I0724 23:38:43.585386    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nspfm_35f71564-7bff-4960-a63b-584726e054eb/kube-rbac-proxy/0.log"