Jul 27 20:49:21.929350 ip-10-0-131-91 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 20:49:21.929364 ip-10-0-131-91 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 20:49:21.929373 ip-10-0-131-91 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 20:49:21.929704 ip-10-0-131-91 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 20:49:31.447166 ip-10-0-131-91 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 20:49:31.447179 ip-10-0-131-91 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2ba655dc818d46849350d0e09d054922 --
Jul 27 20:51:21.219277 ip-10-0-131-91 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 20:51:21.646680 ip-10-0-131-91 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:21.646680 ip-10-0-131-91 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 20:51:21.646680 ip-10-0-131-91 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:21.646680 ip-10-0-131-91 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 20:51:21.646680 ip-10-0-131-91 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:21.647397 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.646756    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 20:51:21.651364 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651348    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651365    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651370    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651374    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651377    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651380    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651383    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651386    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651390    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651394    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651397    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651399    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651402    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651405    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651407    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651410    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651414    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:21.651408 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651417    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651426    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651429    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651436    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651439    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651441    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651449    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651452    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651455    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651457    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651460    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651462    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651465    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651467    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651470    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651472    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651475    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651478    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651480    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:21.651810 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651482    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651485    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651487    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651489    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651492    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651494    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651496    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651499    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651501    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651503    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651506    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651508    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651511    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651513    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651515    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651518    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651521    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651524    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651527    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651530    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:21.652318 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651532    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651535    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651537    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651539    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651542    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651546    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651549    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651552    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651554    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651557    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651559    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651562    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651564    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651566    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651569    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651571    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651573    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651576    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651578    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651581    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:21.652811 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651584    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651586    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651588    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651591    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651593    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651596    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651598    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651601    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651604    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651607    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651610    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651612    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651615    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651617    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651620    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651622    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651624    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651627    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651629    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651632    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651634    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:21.653430 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651636    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:21.654201 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.651639    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:21.654442 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654423    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:21.654472 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654443    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:21.654472 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654450    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:21.654472 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654454    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:21.654472 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654459    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:21.654472 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654463    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:21.654472 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654467    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:21.654472 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654471    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:21.654472 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654475    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654480    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654484    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654486    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654489    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654491    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654494    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654496    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654499    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654501    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654504    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654506    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654509    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654512    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654514    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654517    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654520    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654522    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654525    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654527    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:21.654704 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654530    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654532    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654535    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654538    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654541    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654544    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654546    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654548    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654551    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654554    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654556    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654558    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654561    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654563    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654567    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654570    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654572    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654574    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654578    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654580    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:21.655223 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654583    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654586    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654588    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654591    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654593    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654596    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654600    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654603    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654606    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654609    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654613    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654617    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654620    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654624    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654628    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654631    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654634    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654636    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:21.655706 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654639    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654642    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654644    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654647    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654649    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654652    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654654    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654656    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654660    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654663    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654665    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654668    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654671    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654673    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654677    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654679    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654683    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654686    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654689    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654691    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:21.656162 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654693    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654696    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654699    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654701    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654703    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654706    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654708    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654711    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654714    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654717    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654719    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654722    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.654725    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655169    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655176    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655179    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655184    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655187    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655190    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:21.656647 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655193    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655196    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655199    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655202    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655205    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655207    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655210    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655212    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655215    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655217    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655221    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655223    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655226    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655228    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655231    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655233    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655236    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655239    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655241    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655244    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:21.657118 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655246    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655249    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655251    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655254    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655256    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655259    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655261    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655264    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655266    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655269    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655271    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655274    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655277    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655280    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655283    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655286    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655288    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655291    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655293    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655296    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:21.657595 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655298    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655301    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655303    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655306    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655309    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655311    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655314    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655316    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655318    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655321    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655323    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655326    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655328    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655330    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655333    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655335    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655338    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655341    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655343    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655346    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:21.658078 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655349    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655351    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655354    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655356    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655359    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655362    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655365    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655367    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655370    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655372    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655375    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655377    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655379    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655382    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655384    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655387    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655389    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655393    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655396    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:21.658560 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655399    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655402    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655404    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655407    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655409    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655411    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655414    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655416    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655419    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655422    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655424    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655427    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655429    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.655432    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655506    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655516    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655527    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655534    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655540    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655544    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655551    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655557    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 20:51:21.659024 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655560    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655563    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655566    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655569    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655572    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655575    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655578    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655581    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655584    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655587    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655591    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655594    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655597    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655600    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655603    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655606    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655609    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655612    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655615    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655618    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655621    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655623    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655626    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655629    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 20:51:21.659583 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655633    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655636    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655639    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655641    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655645    2577 flags.go:64] FLAG: --enable-server="true"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655647    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655652    2577 flags.go:64] FLAG: --event-burst="100"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655655    2577 flags.go:64] FLAG: --event-qps="50"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655658    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655661    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655664    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655667    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655670    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655673    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655676    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655678    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655681    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655684    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655687    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655690    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655692    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655695    2577 flags.go:64] FLAG: --feature-gates=""
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655699    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655701    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655705    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 20:51:21.660202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655708    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655711    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655714    2577 flags.go:64] FLAG: --help="false"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655717    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-91.ec2.internal"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655721    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655724    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655726    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655729    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655733    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655735    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655738    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655741    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655744    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655747    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655750    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655753    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655755    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655758    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655761    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655763    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655766    2577 flags.go:64] FLAG: --lock-file=""
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655769    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655772    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655775    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 20:51:21.660799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655780    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655784    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655786    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655789    2577 flags.go:64] FLAG: --logging-format="text"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655792    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655795    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655798    2577 flags.go:64] FLAG: --manifest-url=""
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655801    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655805    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655808    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655812    2577 flags.go:64] FLAG: --max-pods="110"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655815    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655818    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655821    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655823    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655826    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655829    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655832    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655839    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655843    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655846    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655849    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655851    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 20:51:21.661403 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655857    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655859    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655863    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655865    2577 flags.go:64] FLAG: --port="10250"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655869    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655872    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027de7582da9c7533"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655875    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655877    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655880    2577 flags.go:64] FLAG: --register-node="true"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655883    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655886    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655889    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655892    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655895    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655898    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655901    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655904    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655907    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655909    2577 flags.go:64] FLAG: --runonce="false"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655928    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655931    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655934    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655937    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655940    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655943    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655946    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 20:51:21.662025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655949    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655951    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655954    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655957    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655960    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655963    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655966    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655968    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655973    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655976    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655979    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655983    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655985    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655988    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655991    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655994    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.655997    2577 flags.go:64] FLAG: --v="2"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.656000    2577 flags.go:64] FLAG: --version="false"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.656005    2577 flags.go:64] FLAG: --vmodule=""
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.656009    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.656012    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656139    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656143    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656146    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:21.662651 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656149    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656153    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656156    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656159    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656162    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656166    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656169    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656171    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656174    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656177    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656181    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656184    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656186    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656189    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656191    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656195    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656199    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656202    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656205    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656207    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:21.663276 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656210    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656213    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656216    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656218    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656221    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656223    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656226    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656229    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656232    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656238    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656240    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656243    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656245    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656249    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656251    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656254    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656256    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656259    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656261    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656263    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:21.663747 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656266    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656268    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656271    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656273    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656275    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656278    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656280    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656283    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656285    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656288    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656290    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656294    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656297    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656299    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656302    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656305    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656307    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656310    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656312    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:21.664258 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656314    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656317    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656320    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656322    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656325    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656328    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656331    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656334    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656336    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656339    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656341    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656344    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656346    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656349    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656351    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656354    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656356    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656359    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656361    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656364    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:21.664715 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656366    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656368    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656371    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656373    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656376    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656378    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656381    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656383    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656385    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656388    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656390    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656393    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656395    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656397    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656400    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656403    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:21.665209 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.656405    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.656817    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.663475    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.663491    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.663508    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.663517    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663576    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663581    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663585    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663588    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663591    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663594    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663597    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663600    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:21.665592 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663602    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663606    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663610    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663613    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663616    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663619    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663622    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663625    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663628    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663631    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663634    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663636    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663639    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663642    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663645    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663648    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663653    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663656    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663659    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:21.666022 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663661    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663664    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663667    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663670    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663672    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663675    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663677    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663680    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663682    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663685    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663688    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663691    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663693    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663695    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663698    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663700    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663703    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663705    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663708    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:21.666499 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663711    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663713    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663715    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663718    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663720    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663722    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663724    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663727    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663729    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663732    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663734    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663737    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663740    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663742    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663745    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663749    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663752    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663754    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663757    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663760    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:21.667023 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663762    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663765    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663767    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663770    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663772    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663775    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663777    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663779    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663782    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663784    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663787    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663789    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663792    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663794    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663797    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663799    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663803    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663806    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663809    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663811    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:21.667500 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663813    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663816    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663818    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663821    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663824    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663826    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663828    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663832    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663834    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663837    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663839    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663842    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.663844    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.663849    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:21.668070 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664001    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664007    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664010    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664013    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664024    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664027    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664029    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664032    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664034    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664037    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664040    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664042    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664044    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664047    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664050    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664053    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664055    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664058    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664060    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664063    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:21.668619 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664065    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664068    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664070    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664073    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664076    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664079    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664082    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664084    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664087    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664089    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664092    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664094    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664097    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664099    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664102    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664104    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664107    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664109    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664111    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664114    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:21.669403 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664116    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664118    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664121    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664123    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664125    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664128    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664130    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664133    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664136    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664140    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664142    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664145    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664148    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664150    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664153    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664155    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664158    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664160    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664163    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:21.669891 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664165    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664168    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664170    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664173    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664175    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664177    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664180    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664182    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664185    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664187    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664190    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664192    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664194    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664196    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664199    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664201    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664204    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664206    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664208    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:21.670531 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664210    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664214    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664217    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664220    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664222    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664225    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664227    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664229    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664232    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664234    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664237    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664240    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664243    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664245    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664248    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664250    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664253    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664255    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664257    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664260    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:21.671113 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:21.664263    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:21.671851 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.664267    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:21.671851 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.664366    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 20:51:21.671851 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.666340    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 20:51:21.671851 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.668036    2577 server.go:1019] "Starting client certificate rotation"
Jul 27 20:51:21.671851 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.668157    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:51:21.671851 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.668203    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:51:21.671851 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.670389    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:51:21.672181 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.672164    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:51:21.686025 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.686005    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 27 20:51:21.688047 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.688032    2577 log.go:25] "Validated CRI v1 image API"
Jul 27 20:51:21.689138 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.689112    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 20:51:21.693026 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.693005    2577 fs.go:135] Filesystem UUIDs: map[5647d8cd-ebf7-4db4-a5a8-fa3ac73aed06:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 81c17543-f57a-4c97-a0bf-158d8cda4f98:/dev/nvme0n1p4]
Jul 27 20:51:21.693096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.693023    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 20:51:21.697412 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.697390    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:51:21.698055 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.697945    2577 manager.go:216] Machine: {Timestamp:2026-07-27 20:51:21.696670995 +0000 UTC m=+0.349502566 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118240 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec290a3065c434a8355e6dd82ccffc9f SystemUUID:ec290a30-65c4-34a8-355e-6dd82ccffc9f BootID:2ba655dc-818d-4684-9350-d0e09d054922 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c7:9f:19:b0:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c7:9f:19:b0:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:f5:4a:1f:e1:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 20:51:21.698055 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.698052    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 20:51:21.698145 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.698137    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 20:51:21.700962 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.700938    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 20:51:21.701104 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.700965    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 20:51:21.701626 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.701617    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 20:51:21.701669 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.701630    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 20:51:21.701669 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.701642    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:51:21.702839 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.702829    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:51:21.703574 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.703564    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 20:51:21.705180 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705169    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:51:21.705311 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705301    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 20:51:21.705340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705334    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 20:51:21.705370 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705346    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 20:51:21.705370 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705361    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 20:51:21.705370 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705370    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 27 20:51:21.705445 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705379    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 20:51:21.705829 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705817    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:51:21.705901 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.705844    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:51:21.713552 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.713527    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 20:51:21.715298 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.715283    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 20:51:21.715390 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.715320    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 20:51:21.716937 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.716903    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 20:51:21.717013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.716944    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 20:51:21.717013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.716954    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 20:51:21.717013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.716962    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 20:51:21.717013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.716974    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 20:51:21.717013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.716982    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 20:51:21.717013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.716994    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 20:51:21.717013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.717003    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 20:51:21.717013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.717014    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 20:51:21.717354 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.717023    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 20:51:21.717354 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.717037    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 20:51:21.717354 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.717045    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 20:51:21.717354 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.717072    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 20:51:21.717354 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.717080    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 20:51:21.717354 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.717203    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 20:51:21.717607 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.717383    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 20:51:21.719555 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.719529    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9r7vz"
Jul 27 20:51:21.720437 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.720421    2577 server.go:1268] "Started kubelet"
Jul 27 20:51:21.720669 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.720618    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 20:51:21.721423 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.721400    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 20:51:21.721464 ip-10-0-131-91 systemd[1]: Started Kubernetes Kubelet.
Jul 27 20:51:21.721620 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.721533    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 20:51:21.724241 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.724214    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 20:51:21.725854 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.725835    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 20:51:21.726022 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.726003    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9r7vz"
Jul 27 20:51:21.727348 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.727326    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 20:51:21.727659 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.727640    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 20:51:21.732479 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.732463    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 20:51:21.732559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.732486    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 20:51:21.732902 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.732872    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:21.733127 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733112    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 20:51:21.733212 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733203    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 20:51:21.733348 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733117    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 20:51:21.733459 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733441    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 20:51:21.733517 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733460    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 20:51:21.733681 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733657    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 20:51:21.733681 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733681    2577 factory.go:55] Registering systemd factory
Jul 27 20:51:21.733816 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733688    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 27 20:51:21.733946 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733934    2577 factory.go:153] Registering CRI-O factory
Jul 27 20:51:21.734013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733948    2577 factory.go:223] Registration of the crio container factory successfully
Jul 27 20:51:21.734013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733971    2577 factory.go:103] Registering Raw factory
Jul 27 20:51:21.734013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.733984    2577 manager.go:1196] Started watching for new ooms in manager
Jul 27 20:51:21.734727 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.734575    2577 manager.go:349] Starting recovery of all containers
Jul 27 20:51:21.739173 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.739150    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:21.740912 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.740880    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 20:51:21.742976 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.742949    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-91.ec2.internal\" not found" node="ip-10-0-131-91.ec2.internal"
Jul 27 20:51:21.745043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.744969    2577 manager.go:354] Recovery completed
Jul 27 20:51:21.749725 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.749621    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:21.751379 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.751362    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:21.751458 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.751397    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:21.751458 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.751423    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:21.751987 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.751975    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 20:51:21.751987 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.751986    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 20:51:21.752100 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.752002    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:51:21.752100 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.751979    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:21.752698 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.752683    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:21.752742 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.752715    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:21.752742 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.752726    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:21.753314 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.753303    2577 policy_none.go:49] "None policy: Start"
Jul 27 20:51:21.753359 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.753318    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 20:51:21.753359 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.753328    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 20:51:21.754199 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.754188    2577 policy_none.go:47] "Start"
Jul 27 20:51:21.798643 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.798616    2577 manager.go:337] "Starting Device Plugin manager"
Jul 27 20:51:21.798836 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.798678    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 20:51:21.798836 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.798692    2577 server.go:87] "Starting device plugin registration server"
Jul 27 20:51:21.799020 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.799005    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 20:51:21.799077 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.799024    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 20:51:21.799329 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.799151    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 20:51:21.799329 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.799249    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 20:51:21.799329 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.799256    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 20:51:21.799912 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.799879    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 20:51:21.800007 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.799958    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:21.891268 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.891234    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 20:51:21.892173 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.891280    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 20:51:21.892173 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.891303    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 20:51:21.892173 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.891336    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 20:51:21.893292 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.893272    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:21.899594 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.899535    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:21.900349 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.900331    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:21.900430 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.900363    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:21.900430 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.900375    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:21.900430 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.900398    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-91.ec2.internal"
Jul 27 20:51:21.911125 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.911112    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-91.ec2.internal"
Jul 27 20:51:21.911182 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.911133    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-91.ec2.internal\": node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:21.925278 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:21.925255    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:21.992171 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.992126    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"]
Jul 27 20:51:21.992257 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.992215    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:21.993670 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.993654    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:21.993748 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.993684    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:21.993748 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.993694    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:21.994748 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.994736    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:21.994856 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.994844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:21.994886 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.994870    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:21.995449 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.995435    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:21.995513 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.995457    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:21.995513 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.995471    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:21.995586 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.995545    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:21.995586 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.995567    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:21.995586 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.995581    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:21.996652 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.996636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:21.996743 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.996662    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:21.997306 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.997288    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:21.997410 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.997321    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:21.997410 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:21.997334    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:22.017564 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:22.017539    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-91.ec2.internal\" not found" node="ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.022109 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:22.022092    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-91.ec2.internal\" not found" node="ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.026137 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:22.026122    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:22.034831 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.034810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed929be85ebfd2f1b71d7e50ffae918d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-91.ec2.internal\" (UID: \"ed929be85ebfd2f1b71d7e50ffae918d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.034938 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.034840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bf8cac1cb8fccbdc6fd8c2f146bc292-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal\" (UID: \"1bf8cac1cb8fccbdc6fd8c2f146bc292\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.034938 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.034866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf8cac1cb8fccbdc6fd8c2f146bc292-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal\" (UID: \"1bf8cac1cb8fccbdc6fd8c2f146bc292\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.126777 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:22.126748    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:22.135060 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.135039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf8cac1cb8fccbdc6fd8c2f146bc292-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal\" (UID: \"1bf8cac1cb8fccbdc6fd8c2f146bc292\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.135160 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.135066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf8cac1cb8fccbdc6fd8c2f146bc292-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal\" (UID: \"1bf8cac1cb8fccbdc6fd8c2f146bc292\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.135160 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.135076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed929be85ebfd2f1b71d7e50ffae918d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-91.ec2.internal\" (UID: \"ed929be85ebfd2f1b71d7e50ffae918d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.135160 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.135109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed929be85ebfd2f1b71d7e50ffae918d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-91.ec2.internal\" (UID: \"ed929be85ebfd2f1b71d7e50ffae918d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.135160 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.135136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bf8cac1cb8fccbdc6fd8c2f146bc292-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal\" (UID: \"1bf8cac1cb8fccbdc6fd8c2f146bc292\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.135304 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.135169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1bf8cac1cb8fccbdc6fd8c2f146bc292-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal\" (UID: \"1bf8cac1cb8fccbdc6fd8c2f146bc292\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.227574 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:22.227499    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:22.320028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.319996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.324744 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:22.324724    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:22.327682 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:22.327608    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:22.428122 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:22.428068    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:23.815930 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:23.815889    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-91.ec2.internal\" not found"
Jul 27 20:51:23.882798 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.882773    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:23.920828 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.920806    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:23.933049 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.933026    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:51:23.935589 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.935574    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal"
Jul 27 20:51:23.942065 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.942048    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:51:23.954269 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.954246    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 20:51:23.954398 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.954379    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:51:23.954460 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.954413    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:51:23.954460 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.954421    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:51:23.993585 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:23.993557    2577 apiserver.go:52] "Watching apiserver"
Jul 27 20:51:24.009985 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.009964    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 20:51:24.010442 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.010416    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-t8lbq","kube-system/global-pull-secret-syncer-4br5p","kube-system/konnectivity-agent-qqsfw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8","openshift-dns/node-resolver-x5xkb","openshift-network-operator/iptables-alerter-vgvb2","openshift-ovn-kubernetes/ovnkube-node-ztzwl","kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5p4q4","openshift-image-registry/node-ca-ph9gv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal","openshift-multus/multus-additional-cni-plugins-jh5v7","openshift-multus/multus-dx9jr","openshift-multus/network-metrics-daemon-wmbds"]
Jul 27 20:51:24.011656 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.011638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:24.011734 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.011708    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:24.012576 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.012557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.012644 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.012607    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:24.013705 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.013686    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:24.014599 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.014581    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.015726 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.015699    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.017956 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.017471    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 20:51:24.017956 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.017485    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 20:51:24.017956 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.017476    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 20:51:24.017956 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.017805    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 20:51:24.017956 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.017875    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gtnqd\""
Jul 27 20:51:24.018287 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.018041    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:46:23 +0000 UTC" deadline="2028-01-08 17:41:26.291526657 +0000 UTC"
Jul 27 20:51:24.018287 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.018078    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12716h50m2.273453115s"
Jul 27 20:51:24.018287 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.018116    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z7xs7\""
Jul 27 20:51:24.018684 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.018656    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 20:51:24.018684 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.018685    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 20:51:24.019069 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.019034    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xrs2s\""
Jul 27 20:51:24.019305 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.019286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.019383 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.019296    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 20:51:24.019827 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.019807    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 20:51:24.020867 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.020838    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.021472 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.021454    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:51:24.021632 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.021610    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 20:51:24.021743 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.021722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vkptk\""
Jul 27 20:51:24.021831 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.021788    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 20:51:24.022264 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.022250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.023141 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.023124    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nwqx2\""
Jul 27 20:51:24.023286 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.023125    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 20:51:24.023545 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.023527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 20:51:24.023624 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.023571    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.024240 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.024140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 20:51:24.024330 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.024255    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 20:51:24.024330 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.024304    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 20:51:24.024419 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.024309    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 20:51:24.024835 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.024819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.024976 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.024962    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4zr2m\""
Jul 27 20:51:24.025059 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.024979    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:51:24.025121 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-device-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.025171 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqsl8\" (UniqueName: \"kubernetes.io/projected/baca7fdd-b879-41e4-8ac8-b20555a56bb3-kube-api-access-fqsl8\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.025171 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-systemd\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.025293 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025197    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 20:51:24.025293 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-lib-modules\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.025293 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-host\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.025293 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-dbus\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.025475 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-slash\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025475 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-etc-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025475 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-run-netns\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025475 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-systemd\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025475 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-var-lib-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025475 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025475 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-log-socket\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-cni-bin\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-run\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.025729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-iptables-alerter-script\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.025729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-ovnkube-config\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.025729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/82f64a3a-a74e-42bc-ac68-9f087db03925-agent-certs\") pod \"konnectivity-agent-qqsfw\" (UID: \"82f64a3a-a74e-42bc-ac68-9f087db03925\") " pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:24.025729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-sys-fs\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-kubernetes\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysctl-d\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-systemd-units\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-modprobe-d\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jnq\" (UniqueName: \"kubernetes.io/projected/28d16701-b80e-40df-b884-57067b79d323-kube-api-access-m5jnq\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-kubelet-config\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/214751b9-5778-4f01-8602-5b89512a8384-hosts-file\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/214751b9-5778-4f01-8602-5b89512a8384-tmp-dir\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc2jk\" (UniqueName: \"kubernetes.io/projected/214751b9-5778-4f01-8602-5b89512a8384-kube-api-access-vc2jk\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-host-slash\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5g8b\" (UniqueName: \"kubernetes.io/projected/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-kube-api-access-t5g8b\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.026009 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.025987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-kubelet\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-env-overrides\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/82f64a3a-a74e-42bc-ac68-9f087db03925-konnectivity-ca\") pod \"konnectivity-agent-qqsfw\" (UID: \"82f64a3a-a74e-42bc-ac68-9f087db03925\") " pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysconfig\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-ovn\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026288    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026338    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026450    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-socket-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysctl-conf\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28d16701-b80e-40df-b884-57067b79d323-etc-tuned\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.026554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d16701-b80e-40df-b884-57067b79d323-tmp\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-cni-netd\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-ovnkube-script-lib\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jcf\" (UniqueName: \"kubernetes.io/projected/0f632855-5cba-4796-969f-22789c3f505c-kube-api-access-l6jcf\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026677    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-registration-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026688    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tr2bm\""
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-sys\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-var-lib-kubelet\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-node-log\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.026818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f632855-5cba-4796-969f-22789c3f505c-ovn-node-metrics-cert\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.027123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.027047    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 20:51:24.027635 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.027491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:24.027635 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.027535    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vjpkp\""
Jul 27 20:51:24.027635 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.027563    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:24.027736 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.027719    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 20:51:24.028125 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.028111    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 20:51:24.028339 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.028324    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 20:51:24.028425 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.028375    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 20:51:24.028657 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.028644    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c7lql\""
Jul 27 20:51:24.030730 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.030708    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 20:51:24.038530 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.038509    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:51:24.057803 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.057781    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzz69"
Jul 27 20:51:24.064000 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.062299    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:24.067035 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.067015    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bzz69"
Jul 27 20:51:24.120361 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.120331    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded929be85ebfd2f1b71d7e50ffae918d.slice/crio-5787e430a44364b59f146bc4ba7ba00c91b090a62cbc79efc2172eb9aa65a6dd WatchSource:0}: Error finding container 5787e430a44364b59f146bc4ba7ba00c91b090a62cbc79efc2172eb9aa65a6dd: Status 404 returned error can't find the container with id 5787e430a44364b59f146bc4ba7ba00c91b090a62cbc79efc2172eb9aa65a6dd
Jul 27 20:51:24.120582 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.120558    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf8cac1cb8fccbdc6fd8c2f146bc292.slice/crio-93ddf5fdb1a20d81ff2dce72668073887a34f48775214a5dab6f58b3642ecc04 WatchSource:0}: Error finding container 93ddf5fdb1a20d81ff2dce72668073887a34f48775214a5dab6f58b3642ecc04: Status 404 returned error can't find the container with id 93ddf5fdb1a20d81ff2dce72668073887a34f48775214a5dab6f58b3642ecc04
Jul 27 20:51:24.121532 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.121516    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 20:51:24.126012 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.125997    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:51:24.126990 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.126963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-kubelet-config\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.127080 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.127080 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-env-overrides\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.127080 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysconfig\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.127246 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/739950fc-859d-4145-b60c-b5dc0811d195-serviceca\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.127246 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-kubelet-config\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.127246 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.127246 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.127246 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysconfig\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.127246 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-ovn\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.127246 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysctl-conf\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.127542 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-ovn\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.127542 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28d16701-b80e-40df-b884-57067b79d323-etc-tuned\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.127542 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysctl-conf\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.127542 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-cni-bin\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.127542 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-kubelet\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwwq\" (UniqueName: \"kubernetes.io/projected/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-kube-api-access-tfwwq\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-cni-netd\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127620    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127633    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-cni-netd\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-env-overrides\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jcf\" (UniqueName: \"kubernetes.io/projected/0f632855-5cba-4796-969f-22789c3f505c-kube-api-access-l6jcf\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-registration-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-registration-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-var-lib-kubelet\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.127840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f632855-5cba-4796-969f-22789c3f505c-ovn-node-metrics-cert\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127853    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-device-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-etc-selinux\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqsl8\" (UniqueName: \"kubernetes.io/projected/baca7fdd-b879-41e4-8ac8-b20555a56bb3-kube-api-access-fqsl8\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-device-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-var-lib-kubelet\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-lib-modules\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-hostroot\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.127987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-cni-multus\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-systemd\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-log-socket\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-cni-bin\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-lib-modules\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/214751b9-5778-4f01-8602-5b89512a8384-hosts-file\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/214751b9-5778-4f01-8602-5b89512a8384-tmp-dir\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-log-socket\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-system-cni-dir\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.128367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128209    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/214751b9-5778-4f01-8602-5b89512a8384-hosts-file\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-os-release\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128139    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-systemd\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128239    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-cni-bin\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-iptables-alerter-script\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/82f64a3a-a74e-42bc-ac68-9f087db03925-agent-certs\") pod \"konnectivity-agent-qqsfw\" (UID: \"82f64a3a-a74e-42bc-ac68-9f087db03925\") " pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc2jk\" (UniqueName: \"kubernetes.io/projected/214751b9-5778-4f01-8602-5b89512a8384-kube-api-access-vc2jk\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-system-cni-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-k8s-cni-cncf-io\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-systemd-units\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128432    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-multus-certs\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-etc-kubernetes\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/214751b9-5778-4f01-8602-5b89512a8384-tmp-dir\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d16701-b80e-40df-b884-57067b79d323-tmp\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-host-slash\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.128644    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:24.129466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128676    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5g8b\" (UniqueName: \"kubernetes.io/projected/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-kube-api-access-t5g8b\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.128717    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret podName:59a571a8-5d4f-4cb9-8010-f2257b46dbd3 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:24.628681991 +0000 UTC m=+1.994247820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret") pod "global-pull-secret-syncer-4br5p" (UID: "59a571a8-5d4f-4cb9-8010-f2257b46dbd3") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-kubelet\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-host-slash\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-systemd-units\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.128997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/82f64a3a-a74e-42bc-ac68-9f087db03925-konnectivity-ca\") pod \"konnectivity-agent-qqsfw\" (UID: \"82f64a3a-a74e-42bc-ac68-9f087db03925\") " pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-socket-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-cni-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-ovnkube-script-lib\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-sys\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-node-log\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-systemd\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-host\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.130297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129411    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-dbus\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-os-release\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbc44715-5965-40f9-a35d-495832c62fa3-multus-daemon-config\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr6wz\" (UniqueName: \"kubernetes.io/projected/739950fc-859d-4145-b60c-b5dc0811d195-kube-api-access-gr6wz\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-iptables-alerter-script\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cnibin\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-slash\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-etc-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-conf-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-dbus\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.129819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-socket-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-etc-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-systemd\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfr8q\" (UniqueName: \"kubernetes.io/projected/dbc44715-5965-40f9-a35d-495832c62fa3-kube-api-access-hfr8q\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-sys\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-ovnkube-script-lib\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-node-log\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131028 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130392    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-host\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-kubelet\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f72r\" (UniqueName: \"kubernetes.io/projected/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-kube-api-access-4f72r\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130758    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-run-netns\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/82f64a3a-a74e-42bc-ac68-9f087db03925-konnectivity-ca\") pod \"konnectivity-agent-qqsfw\" (UID: \"82f64a3a-a74e-42bc-ac68-9f087db03925\") " pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-var-lib-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-slash\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-run\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-run-ovn-kubernetes\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-run-netns\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-socket-dir-parent\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-var-lib-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-netns\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.130988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-run-openvswitch\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.131732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-run\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-ovnkube-config\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f632855-5cba-4796-969f-22789c3f505c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-sys-fs\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/28d16701-b80e-40df-b884-57067b79d323-etc-tuned\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f632855-5cba-4796-969f-22789c3f505c-ovn-node-metrics-cert\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-kubernetes\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131300    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/28d16701-b80e-40df-b884-57067b79d323-tmp\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/baca7fdd-b879-41e4-8ac8-b20555a56bb3-sys-fs\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysctl-d\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-cnibin\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbc44715-5965-40f9-a35d-495832c62fa3-cni-binary-copy\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/82f64a3a-a74e-42bc-ac68-9f087db03925-agent-certs\") pod \"konnectivity-agent-qqsfw\" (UID: \"82f64a3a-a74e-42bc-ac68-9f087db03925\") " pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-kubernetes\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/739950fc-859d-4145-b60c-b5dc0811d195-host\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.132559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-modprobe-d\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.133366 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-sysctl-d\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.133366 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jnq\" (UniqueName: \"kubernetes.io/projected/28d16701-b80e-40df-b884-57067b79d323-kube-api-access-m5jnq\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.133366 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/28d16701-b80e-40df-b884-57067b79d323-etc-modprobe-d\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.133366 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.131724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f632855-5cba-4796-969f-22789c3f505c-ovnkube-config\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.137465 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.137444    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:24.137565 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.137470    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:24.137565 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.137483    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4cm6 for pod openshift-network-diagnostics/network-check-target-t8lbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:24.137565 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.137553    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6 podName:ab76bb44-d4bb-467c-9c4b-275f4f60b303 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:24.637535067 +0000 UTC m=+2.003100912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h4cm6" (UniqueName: "kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6") pod "network-check-target-t8lbq" (UID: "ab76bb44-d4bb-467c-9c4b-275f4f60b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:24.137783 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.137765    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqsl8\" (UniqueName: \"kubernetes.io/projected/baca7fdd-b879-41e4-8ac8-b20555a56bb3-kube-api-access-fqsl8\") pod \"aws-ebs-csi-driver-node-66bg8\" (UID: \"baca7fdd-b879-41e4-8ac8-b20555a56bb3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.138389 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.138372    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc2jk\" (UniqueName: \"kubernetes.io/projected/214751b9-5778-4f01-8602-5b89512a8384-kube-api-access-vc2jk\") pod \"node-resolver-x5xkb\" (UID: \"214751b9-5778-4f01-8602-5b89512a8384\") " pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.138867 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.138850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jcf\" (UniqueName: \"kubernetes.io/projected/0f632855-5cba-4796-969f-22789c3f505c-kube-api-access-l6jcf\") pod \"ovnkube-node-ztzwl\" (UID: \"0f632855-5cba-4796-969f-22789c3f505c\") " pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.139393 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.139377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5g8b\" (UniqueName: \"kubernetes.io/projected/5a3a514c-189c-4493-b288-d1ec0d2ce9dd-kube-api-access-t5g8b\") pod \"iptables-alerter-vgvb2\" (UID: \"5a3a514c-189c-4493-b288-d1ec0d2ce9dd\") " pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.142997 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.142980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jnq\" (UniqueName: \"kubernetes.io/projected/28d16701-b80e-40df-b884-57067b79d323-kube-api-access-m5jnq\") pod \"tuned-5p4q4\" (UID: \"28d16701-b80e-40df-b884-57067b79d323\") " pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.182082 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.182031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal" event={"ID":"ed929be85ebfd2f1b71d7e50ffae918d","Type":"ContainerStarted","Data":"5787e430a44364b59f146bc4ba7ba00c91b090a62cbc79efc2172eb9aa65a6dd"}
Jul 27 20:51:24.183202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.183168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal" event={"ID":"1bf8cac1cb8fccbdc6fd8c2f146bc292","Type":"ContainerStarted","Data":"93ddf5fdb1a20d81ff2dce72668073887a34f48775214a5dab6f58b3642ecc04"}
Jul 27 20:51:24.232807 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-system-cni-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.232964 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-k8s-cni-cncf-io\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.232964 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-multus-certs\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.232964 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-multus-certs\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.232964 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-system-cni-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.232964 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-etc-kubernetes\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.232964 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-k8s-cni-cncf-io\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.232964 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232958    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-etc-kubernetes\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.232969    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-cni-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-os-release\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbc44715-5965-40f9-a35d-495832c62fa3-multus-daemon-config\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr6wz\" (UniqueName: \"kubernetes.io/projected/739950fc-859d-4145-b60c-b5dc0811d195-kube-api-access-gr6wz\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-cni-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cnibin\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cnibin\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-conf-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-os-release\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfr8q\" (UniqueName: \"kubernetes.io/projected/dbc44715-5965-40f9-a35d-495832c62fa3-kube-api-access-hfr8q\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f72r\" (UniqueName: \"kubernetes.io/projected/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-kube-api-access-4f72r\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-conf-dir\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-socket-dir-parent\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-netns\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-multus-socket-dir-parent\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:24.233340 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-run-netns\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233332    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-cnibin\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbc44715-5965-40f9-a35d-495832c62fa3-cni-binary-copy\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-cnibin\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/739950fc-859d-4145-b60c-b5dc0811d195-host\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.233435    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/739950fc-859d-4145-b60c-b5dc0811d195-serviceca\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/739950fc-859d-4145-b60c-b5dc0811d195-host\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.233499    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs podName:b322eaf2-56c8-448e-83d0-c4fb746fb0a8 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:24.733481505 +0000 UTC m=+2.099047330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs") pod "network-metrics-daemon-wmbds" (UID: "b322eaf2-56c8-448e-83d0-c4fb746fb0a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-cni-bin\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-kubelet\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwwq\" (UniqueName: \"kubernetes.io/projected/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-kube-api-access-tfwwq\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-hostroot\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-cni-multus\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbc44715-5965-40f9-a35d-495832c62fa3-multus-daemon-config\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-system-cni-dir\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-system-cni-dir\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-kubelet\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-os-release\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-os-release\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233840    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-cni-bin\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbc44715-5965-40f9-a35d-495832c62fa3-cni-binary-copy\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/739950fc-859d-4145-b60c-b5dc0811d195-serviceca\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233923    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-hostroot\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbc44715-5965-40f9-a35d-495832c62fa3-host-var-lib-cni-multus\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-cni-binary-copy\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.234595 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.233954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.242450 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.242424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfr8q\" (UniqueName: \"kubernetes.io/projected/dbc44715-5965-40f9-a35d-495832c62fa3-kube-api-access-hfr8q\") pod \"multus-dx9jr\" (UID: \"dbc44715-5965-40f9-a35d-495832c62fa3\") " pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.242569 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.242426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f72r\" (UniqueName: \"kubernetes.io/projected/af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff-kube-api-access-4f72r\") pod \"multus-additional-cni-plugins-jh5v7\" (UID: \"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff\") " pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.242708 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.242690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwwq\" (UniqueName: \"kubernetes.io/projected/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-kube-api-access-tfwwq\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:24.242766 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.242737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr6wz\" (UniqueName: \"kubernetes.io/projected/739950fc-859d-4145-b60c-b5dc0811d195-kube-api-access-gr6wz\") pod \"node-ca-ph9gv\" (UID: \"739950fc-859d-4145-b60c-b5dc0811d195\") " pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.341885 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.341794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:24.350375 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.350350    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f64a3a_a74e_42bc_ac68_9f087db03925.slice/crio-878edd9112bac58e4df637fb806a745d4e2609fe52e048e4ad463d1e79351170 WatchSource:0}: Error finding container 878edd9112bac58e4df637fb806a745d4e2609fe52e048e4ad463d1e79351170: Status 404 returned error can't find the container with id 878edd9112bac58e4df637fb806a745d4e2609fe52e048e4ad463d1e79351170
Jul 27 20:51:24.356923 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.356902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8"
Jul 27 20:51:24.364626 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.364588    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaca7fdd_b879_41e4_8ac8_b20555a56bb3.slice/crio-ea2adb849f6f3cd8da7df0d0df882fa174289f441f5e88d88a249a9479dd3b8e WatchSource:0}: Error finding container ea2adb849f6f3cd8da7df0d0df882fa174289f441f5e88d88a249a9479dd3b8e: Status 404 returned error can't find the container with id ea2adb849f6f3cd8da7df0d0df882fa174289f441f5e88d88a249a9479dd3b8e
Jul 27 20:51:24.370878 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.370857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x5xkb"
Jul 27 20:51:24.375530 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.375502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vgvb2"
Jul 27 20:51:24.378537 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.378504    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214751b9_5778_4f01_8602_5b89512a8384.slice/crio-ee2fa542ed36ce876dbe5da156897b28199bfbf783ccd08f26f059e0bdbfadfb WatchSource:0}: Error finding container ee2fa542ed36ce876dbe5da156897b28199bfbf783ccd08f26f059e0bdbfadfb: Status 404 returned error can't find the container with id ee2fa542ed36ce876dbe5da156897b28199bfbf783ccd08f26f059e0bdbfadfb
Jul 27 20:51:24.383441 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.383419    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3a514c_189c_4493_b288_d1ec0d2ce9dd.slice/crio-12741932dcb9f1dc4e7fe4de4181fe7c3c7f9641cd0d5ad6df3c0ac2afe94a8d WatchSource:0}: Error finding container 12741932dcb9f1dc4e7fe4de4181fe7c3c7f9641cd0d5ad6df3c0ac2afe94a8d: Status 404 returned error can't find the container with id 12741932dcb9f1dc4e7fe4de4181fe7c3c7f9641cd0d5ad6df3c0ac2afe94a8d
Jul 27 20:51:24.389846 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.389823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:24.397235 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.397210    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f632855_5cba_4796_969f_22789c3f505c.slice/crio-042f6d0c3e6d8ba54af7d28983271844ba8065eb50378b3c773e638ee68fcf32 WatchSource:0}: Error finding container 042f6d0c3e6d8ba54af7d28983271844ba8065eb50378b3c773e638ee68fcf32: Status 404 returned error can't find the container with id 042f6d0c3e6d8ba54af7d28983271844ba8065eb50378b3c773e638ee68fcf32
Jul 27 20:51:24.406957 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.406936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5p4q4"
Jul 27 20:51:24.413654 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.413632    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d16701_b80e_40df_b884_57067b79d323.slice/crio-7ec77671e9f425507b2456b8f3982aeac40780bc940a878f75840c4e093b1471 WatchSource:0}: Error finding container 7ec77671e9f425507b2456b8f3982aeac40780bc940a878f75840c4e093b1471: Status 404 returned error can't find the container with id 7ec77671e9f425507b2456b8f3982aeac40780bc940a878f75840c4e093b1471
Jul 27 20:51:24.421873 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.421853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ph9gv"
Jul 27 20:51:24.427512 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.427492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jh5v7"
Jul 27 20:51:24.429880 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.429858    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739950fc_859d_4145_b60c_b5dc0811d195.slice/crio-789ded8ec1ed9dcad08a85d04f57999478d88cf5800505cbc8b7b66f4c9742a7 WatchSource:0}: Error finding container 789ded8ec1ed9dcad08a85d04f57999478d88cf5800505cbc8b7b66f4c9742a7: Status 404 returned error can't find the container with id 789ded8ec1ed9dcad08a85d04f57999478d88cf5800505cbc8b7b66f4c9742a7
Jul 27 20:51:24.434805 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:24.434781    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3fea87_92d3_4b71_b2a0_edf0b0b1f9ff.slice/crio-4937ba4c9e800ddc65ccf42b7eb2978c38d017d7cf1184e34f058057e05af093 WatchSource:0}: Error finding container 4937ba4c9e800ddc65ccf42b7eb2978c38d017d7cf1184e34f058057e05af093: Status 404 returned error can't find the container with id 4937ba4c9e800ddc65ccf42b7eb2978c38d017d7cf1184e34f058057e05af093
Jul 27 20:51:24.435798 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.435782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dx9jr"
Jul 27 20:51:24.637010 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.636934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:24.637168 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.637116    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:24.637253 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.637174    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret podName:59a571a8-5d4f-4cb9-8010-f2257b46dbd3 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:25.637156246 +0000 UTC m=+3.002722074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret") pod "global-pull-secret-syncer-4br5p" (UID: "59a571a8-5d4f-4cb9-8010-f2257b46dbd3") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:24.737417 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.737380    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:24.737602 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.737514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:24.737673 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.737598    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:24.737720 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.737700    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs podName:b322eaf2-56c8-448e-83d0-c4fb746fb0a8 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:25.737679981 +0000 UTC m=+3.103245813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs") pod "network-metrics-daemon-wmbds" (UID: "b322eaf2-56c8-448e-83d0-c4fb746fb0a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:24.738113 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.738093    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:24.738113 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.738115    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:24.738309 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.738128    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4cm6 for pod openshift-network-diagnostics/network-check-target-t8lbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:24.738309 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:24.738229    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6 podName:ab76bb44-d4bb-467c-9c4b-275f4f60b303 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:25.738178359 +0000 UTC m=+3.103744186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4cm6" (UniqueName: "kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6") pod "network-check-target-t8lbq" (UID: "ab76bb44-d4bb-467c-9c4b-275f4f60b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:24.919057 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:24.918956    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:25.063509 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.063288    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:25.069558 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.069493    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:46:24 +0000 UTC" deadline="2028-02-26 02:02:21.811411346 +0000 UTC"
Jul 27 20:51:25.069558 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.069558    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13877h10m56.741858161s"
Jul 27 20:51:25.179359 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.179330    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:25.179502 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.179471    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:25.179564 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.179527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:25.179677 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.179650    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:25.207661 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.207603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dx9jr" event={"ID":"dbc44715-5965-40f9-a35d-495832c62fa3","Type":"ContainerStarted","Data":"c1ea5e9f3eb3bca06f32f263020041d342a0718ef78282dd14174e9b2aab61a0"}
Jul 27 20:51:25.221705 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.221621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ph9gv" event={"ID":"739950fc-859d-4145-b60c-b5dc0811d195","Type":"ContainerStarted","Data":"789ded8ec1ed9dcad08a85d04f57999478d88cf5800505cbc8b7b66f4c9742a7"}
Jul 27 20:51:25.231945 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.231837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5p4q4" event={"ID":"28d16701-b80e-40df-b884-57067b79d323","Type":"ContainerStarted","Data":"7ec77671e9f425507b2456b8f3982aeac40780bc940a878f75840c4e093b1471"}
Jul 27 20:51:25.244123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.243724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"042f6d0c3e6d8ba54af7d28983271844ba8065eb50378b3c773e638ee68fcf32"}
Jul 27 20:51:25.254463 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.254379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgvb2" event={"ID":"5a3a514c-189c-4493-b288-d1ec0d2ce9dd","Type":"ContainerStarted","Data":"12741932dcb9f1dc4e7fe4de4181fe7c3c7f9641cd0d5ad6df3c0ac2afe94a8d"}
Jul 27 20:51:25.259447 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.259414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x5xkb" event={"ID":"214751b9-5778-4f01-8602-5b89512a8384","Type":"ContainerStarted","Data":"ee2fa542ed36ce876dbe5da156897b28199bfbf783ccd08f26f059e0bdbfadfb"}
Jul 27 20:51:25.272224 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.272171    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8" event={"ID":"baca7fdd-b879-41e4-8ac8-b20555a56bb3","Type":"ContainerStarted","Data":"ea2adb849f6f3cd8da7df0d0df882fa174289f441f5e88d88a249a9479dd3b8e"}
Jul 27 20:51:25.292105 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.291998    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" event={"ID":"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff","Type":"ContainerStarted","Data":"4937ba4c9e800ddc65ccf42b7eb2978c38d017d7cf1184e34f058057e05af093"}
Jul 27 20:51:25.296757 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.296725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qqsfw" event={"ID":"82f64a3a-a74e-42bc-ac68-9f087db03925","Type":"ContainerStarted","Data":"878edd9112bac58e4df637fb806a745d4e2609fe52e048e4ad463d1e79351170"}
Jul 27 20:51:25.647956 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.647366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:25.647956 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.647533    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:25.647956 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.647599    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret podName:59a571a8-5d4f-4cb9-8010-f2257b46dbd3 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:27.647579079 +0000 UTC m=+5.013144905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret") pod "global-pull-secret-syncer-4br5p" (UID: "59a571a8-5d4f-4cb9-8010-f2257b46dbd3") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:25.748044 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.747996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:25.748270 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:25.748071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:25.748270 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.748230    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:25.748270 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.748232    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:25.748270 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.748259    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:25.748270 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.748271    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4cm6 for pod openshift-network-diagnostics/network-check-target-t8lbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:25.748545 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.748298    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs podName:b322eaf2-56c8-448e-83d0-c4fb746fb0a8 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:27.748277657 +0000 UTC m=+5.113843492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs") pod "network-metrics-daemon-wmbds" (UID: "b322eaf2-56c8-448e-83d0-c4fb746fb0a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:25.748545 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:25.748324    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6 podName:ab76bb44-d4bb-467c-9c4b-275f4f60b303 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:27.748305448 +0000 UTC m=+5.113871279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4cm6" (UniqueName: "kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6") pod "network-check-target-t8lbq" (UID: "ab76bb44-d4bb-467c-9c4b-275f4f60b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:26.070328 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:26.070276    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:46:24 +0000 UTC" deadline="2028-03-29 12:16:48.919845324 +0000 UTC"
Jul 27 20:51:26.070328 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:26.070315    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14655h25m22.849535041s"
Jul 27 20:51:26.178821 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:26.178777    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:26.179011 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:26.178931    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:26.558033 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:26.557806    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:27.180793 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:27.180754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:27.181311 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.180876    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:27.181311 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:27.181163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:27.181311 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.181247    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:27.662069 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:27.661994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:27.662293 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.662210    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:27.662293 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.662280    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret podName:59a571a8-5d4f-4cb9-8010-f2257b46dbd3 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:31.662260654 +0000 UTC m=+9.027826503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret") pod "global-pull-secret-syncer-4br5p" (UID: "59a571a8-5d4f-4cb9-8010-f2257b46dbd3") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:27.764068 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:27.763128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:27.764068 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:27.763204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:27.764068 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.763374    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:27.764068 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.763437    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs podName:b322eaf2-56c8-448e-83d0-c4fb746fb0a8 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:31.763416601 +0000 UTC m=+9.128982443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs") pod "network-metrics-daemon-wmbds" (UID: "b322eaf2-56c8-448e-83d0-c4fb746fb0a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:27.764068 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.763957    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:27.764068 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.763978    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:27.764068 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.763991    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4cm6 for pod openshift-network-diagnostics/network-check-target-t8lbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:27.764068 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:27.764035    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6 podName:ab76bb44-d4bb-467c-9c4b-275f4f60b303 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:31.764019507 +0000 UTC m=+9.129585345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4cm6" (UniqueName: "kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6") pod "network-check-target-t8lbq" (UID: "ab76bb44-d4bb-467c-9c4b-275f4f60b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:28.179758 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:28.179225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:28.179758 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:28.179381    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:29.179111 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:29.179077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:29.179534 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:29.179080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:29.179534 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:29.179229    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:29.179534 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:29.179288    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:30.179700 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:30.179665    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:30.180173 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:30.179821    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:31.179540 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:31.179511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:31.179712 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.179614    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:31.180029 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:31.180005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:31.180440 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.180097    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:31.696357 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:31.696317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:31.696534 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.696516    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:31.696591 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.696582    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret podName:59a571a8-5d4f-4cb9-8010-f2257b46dbd3 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:39.696561597 +0000 UTC m=+17.062127424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret") pod "global-pull-secret-syncer-4br5p" (UID: "59a571a8-5d4f-4cb9-8010-f2257b46dbd3") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:31.796924 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:31.796886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:31.797089 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:31.796979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:31.797171 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.797123    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:31.797171 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.797121    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:31.797171 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.797143    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:31.797171 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.797156    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4cm6 for pod openshift-network-diagnostics/network-check-target-t8lbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:31.797396 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.797212    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs podName:b322eaf2-56c8-448e-83d0-c4fb746fb0a8 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:39.797173751 +0000 UTC m=+17.162739580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs") pod "network-metrics-daemon-wmbds" (UID: "b322eaf2-56c8-448e-83d0-c4fb746fb0a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:31.797396 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:31.797231    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6 podName:ab76bb44-d4bb-467c-9c4b-275f4f60b303 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:39.797221746 +0000 UTC m=+17.162787571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4cm6" (UniqueName: "kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6") pod "network-check-target-t8lbq" (UID: "ab76bb44-d4bb-467c-9c4b-275f4f60b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:32.179988 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:32.179386    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:32.179988 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:32.179550    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:33.181060 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:33.181018    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:33.181532 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:33.181156    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:33.181594 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:33.181023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:33.181642 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:33.181628    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:34.179683 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:34.179594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:34.179845 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:34.179781    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:35.179367 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:35.179306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:35.179813 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:35.179442    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:35.179813 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:35.179482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:35.179813 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:35.179581    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:36.179478 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:36.179392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:36.179865 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:36.179562    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:37.179349 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:37.179129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:37.179533 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:37.179205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:37.179533 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:37.179428    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:37.179533 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:37.179492    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:38.179695 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:38.179660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:38.180108 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:38.179789    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:39.182465 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:39.182020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:39.182465 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:39.182030    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:39.182465 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.182123    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:39.182465 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.182242    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:39.764373 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:39.764337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:39.764563 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.764459    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:39.764563 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.764527    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret podName:59a571a8-5d4f-4cb9-8010-f2257b46dbd3 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:55.764506421 +0000 UTC m=+33.130072251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret") pod "global-pull-secret-syncer-4br5p" (UID: "59a571a8-5d4f-4cb9-8010-f2257b46dbd3") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:39.865604 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:39.865577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:39.865742 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:39.865619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:39.865742 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.865726    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:39.865816 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.865743    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:39.865816 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.865760    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:39.865816 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.865773    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4cm6 for pod openshift-network-diagnostics/network-check-target-t8lbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:39.865816 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.865790    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs podName:b322eaf2-56c8-448e-83d0-c4fb746fb0a8 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:55.865772183 +0000 UTC m=+33.231338014 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs") pod "network-metrics-daemon-wmbds" (UID: "b322eaf2-56c8-448e-83d0-c4fb746fb0a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:39.865816 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:39.865806    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6 podName:ab76bb44-d4bb-467c-9c4b-275f4f60b303 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:55.865797033 +0000 UTC m=+33.231362858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4cm6" (UniqueName: "kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6") pod "network-check-target-t8lbq" (UID: "ab76bb44-d4bb-467c-9c4b-275f4f60b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:40.179527 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:40.179498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:40.179685 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:40.179622    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:41.179290 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:41.179250    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:41.179740 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:41.179395    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:41.179740 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:41.179433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:41.179740 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:41.179518    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:42.179622 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:42.179584    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:42.180031 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:42.179737    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:43.180674 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:43.180636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:43.181111 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:43.180757    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:43.181111 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:43.180799    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:43.181111 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:43.180904    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:44.179233 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:44.179210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:44.179340 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:44.179320    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:44.356999 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:44.355029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dx9jr" event={"ID":"dbc44715-5965-40f9-a35d-495832c62fa3","Type":"ContainerStarted","Data":"0a2f6492575ca6698372a64140515a9dfada34011462ea14b45c78dd6594c336"}
Jul 27 20:51:44.361757 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:44.361722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5p4q4" event={"ID":"28d16701-b80e-40df-b884-57067b79d323","Type":"ContainerStarted","Data":"8c36f6fc2b523625ffbc3154d94a01bf643ed2d72fbe6f23fd4739be8a782fef"}
Jul 27 20:51:44.366797 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:44.366281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal" event={"ID":"ed929be85ebfd2f1b71d7e50ffae918d","Type":"ContainerStarted","Data":"0d38a80cfd8ba36d4b78788c1c4de8bd832017dc98dd93a60f7a3d6637406f48"}
Jul 27 20:51:44.373495 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:44.373047    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dx9jr" podStartSLOduration=1.8625624379999999 podStartE2EDuration="21.373030088s" totalImagesPullingTime="19.51046765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:44.372825548 +0000 UTC m=+21.738391396" watchObservedRunningTime="2026-07-27 20:51:44.373030088 +0000 UTC m=+21.738595946"
Jul 27 20:51:44.417230 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:44.416765    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5p4q4" podStartSLOduration=1.89181197 podStartE2EDuration="21.416741846s" totalImagesPullingTime="19.524929876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:44.40186298 +0000 UTC m=+21.767428828" watchObservedRunningTime="2026-07-27 20:51:44.416741846 +0000 UTC m=+21.782307694"
Jul 27 20:51:45.179424 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.178924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:45.179424 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.178963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:45.179609 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:45.179458    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:45.179609 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:45.179537    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:45.369550 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.369514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8" event={"ID":"baca7fdd-b879-41e4-8ac8-b20555a56bb3","Type":"ContainerStarted","Data":"a4cf16d904aba3fcfb112af1553a154d9462c9aa6659d8a9634d594380f12060"}
Jul 27 20:51:45.370963 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.370939    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff" containerID="b6800d935be5722dbd569ae9976a696d99eda5db43a43e9a678c2de8c335c2c5" exitCode=0
Jul 27 20:51:45.371082 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.371007    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" event={"ID":"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff","Type":"ContainerDied","Data":"b6800d935be5722dbd569ae9976a696d99eda5db43a43e9a678c2de8c335c2c5"}
Jul 27 20:51:45.372601 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.372565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qqsfw" event={"ID":"82f64a3a-a74e-42bc-ac68-9f087db03925","Type":"ContainerStarted","Data":"57c837f798f57085a88a1171bc479746160260cf4098a610da28fc2033a38980"}
Jul 27 20:51:45.374035 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.374006    2577 generic.go:356] "Generic (PLEG): container finished" podID="1bf8cac1cb8fccbdc6fd8c2f146bc292" containerID="ffeb43c6a38ea00e79d50859b5616ebd6ff4b0b58524c2fde56884c16d53d9e1" exitCode=0
Jul 27 20:51:45.374147 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.374088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal" event={"ID":"1bf8cac1cb8fccbdc6fd8c2f146bc292","Type":"ContainerDied","Data":"ffeb43c6a38ea00e79d50859b5616ebd6ff4b0b58524c2fde56884c16d53d9e1"}
Jul 27 20:51:45.383154 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.383124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ph9gv" event={"ID":"739950fc-859d-4145-b60c-b5dc0811d195","Type":"ContainerStarted","Data":"c4f2863c2d6240b2feb8fd5951d8b6c26cfec7d837ff273c756f3383e86ef378"}
Jul 27 20:51:45.386026 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.385926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"c731b2858bdff4d75dd3fa83cbfb39c1b96dfc0556e3dc13d3f494c926cda086"}
Jul 27 20:51:45.386026 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.385979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"432c8736629519b4dd3526ab21d6398730b4bdb833adcef8c77df9544c8c90a0"}
Jul 27 20:51:45.386026 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.386022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"775a6fad4f89b68770903c9418dc7e814539f51ed9491781e678725efe46a7f3"}
Jul 27 20:51:45.386204 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.386055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"4285b8a4930e6538bda96d8b10330c35c53c027880f30b4c0a63d20b9bb6c4b4"}
Jul 27 20:51:45.386204 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.386085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"d7862d31cfecb73eb8b7d427a200c84d86a4f83714b6cce8508dde43544e4c0d"}
Jul 27 20:51:45.386204 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.386117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"a007e9239da0e52ae4490ba53142b030386a0e6f59ea22726ad288dd6a504a7f"}
Jul 27 20:51:45.387160 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.387143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x5xkb" event={"ID":"214751b9-5778-4f01-8602-5b89512a8384","Type":"ContainerStarted","Data":"9e0d3c83dac287dad586fa7a0eeb6b160ede9ee2ac57897c74fb705b54407dbc"}
Jul 27 20:51:45.402474 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.402430    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-91.ec2.internal" podStartSLOduration=22.402415196 podStartE2EDuration="22.402415196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:44.417037184 +0000 UTC m=+21.782603031" watchObservedRunningTime="2026-07-27 20:51:45.402415196 +0000 UTC m=+22.767981034"
Jul 27 20:51:45.431013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.430930    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qqsfw" podStartSLOduration=2.863676678 podStartE2EDuration="22.430904224s" totalImagesPullingTime="19.567227546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:45.417058572 +0000 UTC m=+22.782624420" watchObservedRunningTime="2026-07-27 20:51:45.430904224 +0000 UTC m=+22.796470049"
Jul 27 20:51:45.431406 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.431380    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ph9gv" podStartSLOduration=2.94358847 podStartE2EDuration="22.431373205s" totalImagesPullingTime="19.487784735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:45.43105291 +0000 UTC m=+22.796618739" watchObservedRunningTime="2026-07-27 20:51:45.431373205 +0000 UTC m=+22.796939051"
Jul 27 20:51:45.460681 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:45.460629    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x5xkb" podStartSLOduration=2.921322999 podStartE2EDuration="22.460611585s" totalImagesPullingTime="19.539288586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:45.460307641 +0000 UTC m=+22.825873490" watchObservedRunningTime="2026-07-27 20:51:45.460611585 +0000 UTC m=+22.826177457"
Jul 27 20:51:46.029434 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.029406    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 20:51:46.100175 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.100067    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T20:51:46.029428174Z","UUID":"2648998d-d9b0-48fa-bb89-c5ad9a03e435","Handler":null,"Name":"","Endpoint":""}
Jul 27 20:51:46.101617 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.101594    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 20:51:46.101724 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.101638    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 20:51:46.179047 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.179017    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:46.179269 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:46.179134    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:46.390792 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.390707    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vgvb2" event={"ID":"5a3a514c-189c-4493-b288-d1ec0d2ce9dd","Type":"ContainerStarted","Data":"3bfc8258cfc52e805738037d374211279f2d2e128f60b3273e626f405cb816d1"}
Jul 27 20:51:46.392657 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.392632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8" event={"ID":"baca7fdd-b879-41e4-8ac8-b20555a56bb3","Type":"ContainerStarted","Data":"ad0e693ca90a52e2b0b2fcbc9658fc2aea009df5646b86b0d26db2b8f44d642f"}
Jul 27 20:51:46.394598 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.394563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal" event={"ID":"1bf8cac1cb8fccbdc6fd8c2f146bc292","Type":"ContainerStarted","Data":"23d5790d1a45f93623355a0d8c2d5b3664357ada8b342231bf61819243caffde"}
Jul 27 20:51:46.406091 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.406042    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vgvb2" podStartSLOduration=3.847314156 podStartE2EDuration="23.406027849s" totalImagesPullingTime="19.558713693s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:46.405823324 +0000 UTC m=+23.771389172" watchObservedRunningTime="2026-07-27 20:51:46.406027849 +0000 UTC m=+23.771593695"
Jul 27 20:51:46.422690 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:46.422645    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-91.ec2.internal" podStartSLOduration=23.422631124 podStartE2EDuration="23.422631124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:46.421709816 +0000 UTC m=+23.787275663" watchObservedRunningTime="2026-07-27 20:51:46.422631124 +0000 UTC m=+23.788196972"
Jul 27 20:51:47.178843 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:47.178816    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:47.178957 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:47.178931    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:47.179243 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:47.178819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:47.179340 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:47.179318    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:47.400312 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:47.400230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8" event={"ID":"baca7fdd-b879-41e4-8ac8-b20555a56bb3","Type":"ContainerStarted","Data":"1a4e978824bb9214139c272693eacb31350a1b9972e1530694491fb296ffcc37"}
Jul 27 20:51:47.404175 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:47.404132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"7ef750060b9f6c0acc403681e3acae3f3187e2f57ced0b5b48eed1ce7889c9ee"}
Jul 27 20:51:47.419073 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:47.419019    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-66bg8" podStartSLOduration=2.488700135 podStartE2EDuration="24.419003241s" totalImagesPullingTime="21.930303106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:47.418584671 +0000 UTC m=+24.784150519" watchObservedRunningTime="2026-07-27 20:51:47.419003241 +0000 UTC m=+24.784569081"
Jul 27 20:51:48.179755 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:48.179718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:48.179939 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:48.179870    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:49.178857 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:49.178815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:49.179339 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:49.178947    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:49.179339 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:49.179001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:49.179339 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:49.179110    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:49.825078 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:49.824903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:49.917223 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:49.917171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:49.984417 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:49.984387    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:50.179145 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.179081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:50.179676 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:50.179199    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:50.271055 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.271026    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:50.271618 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.271600    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:50.414324 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.414288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" event={"ID":"0f632855-5cba-4796-969f-22789c3f505c","Type":"ContainerStarted","Data":"e77fbae3520780656e732fa091ba443c1df9e45675f7b0b4e2845717be9465da"}
Jul 27 20:51:50.414554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.414532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:50.414649 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.414608    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:50.416205 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.416163    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff" containerID="055dacc72a0b34ebba625e0b5cb6301df82b4b309f216ac7a78a820c808605df" exitCode=0
Jul 27 20:51:50.416309 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.416216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" event={"ID":"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff","Type":"ContainerDied","Data":"055dacc72a0b34ebba625e0b5cb6301df82b4b309f216ac7a78a820c808605df"}
Jul 27 20:51:50.423834 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.423814    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:50.430844 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.430790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:50.445775 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.445737    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl" podStartSLOduration=7.50755485 podStartE2EDuration="27.44572358s" totalImagesPullingTime="19.93816873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:50.445289543 +0000 UTC m=+27.810855390" watchObservedRunningTime="2026-07-27 20:51:50.44572358 +0000 UTC m=+27.811289427"
Jul 27 20:51:50.502342 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.502317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:50.570901 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:50.570872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:51.183580 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.183337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:51.183993 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.183359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:51.183993 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:51.183650    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:51.183993 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:51.183749    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:51.235824 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.235793    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:51.236684 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.236649    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qqsfw"
Jul 27 20:51:51.309468 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.309443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:51.412880 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.412839    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8lbq"]
Jul 27 20:51:51.413686 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.413663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wmbds"]
Jul 27 20:51:51.413806 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.413795    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:51.413946 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:51.413915    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:51.414328 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.414302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4br5p"]
Jul 27 20:51:51.420937 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.420875    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff" containerID="a82f96701d0938ad88cb0398936cfaf861414353bd1359edccaa647563063386" exitCode=0
Jul 27 20:51:51.421046 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.420949    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:51.421046 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.420981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" event={"ID":"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff","Type":"ContainerDied","Data":"a82f96701d0938ad88cb0398936cfaf861414353bd1359edccaa647563063386"}
Jul 27 20:51:51.421154 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:51.421052    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:51.422297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.421628    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:51.422297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.421877    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:51.422297 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:51.421975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:51.440363 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.440340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:51:51.679638 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.679560    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:51.736645 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:51.736579    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:52.425454 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:52.425422    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff" containerID="5708f388243cfc6a36684b6331e9887f88f498c73b7de91db37f01e92a1b8770" exitCode=0
Jul 27 20:51:52.425843 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:52.425512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" event={"ID":"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff","Type":"ContainerDied","Data":"5708f388243cfc6a36684b6331e9887f88f498c73b7de91db37f01e92a1b8770"}
Jul 27 20:51:53.182742 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:53.182710    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:53.182926 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:53.182820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:53.182926 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:53.182832    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:53.182926 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:53.182914    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:53.183061 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:53.182813    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:53.183061 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:53.183020    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:53.600451 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:53.600402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:53.697385 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:53.697288    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:54.531946 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:54.531912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:54.600780 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:54.600750    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:54.668196 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:54.668163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:54.778481 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:54.778451    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:54.852919 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:54.852677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:55.179469 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:55.179388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:55.179643 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:55.179388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:55.179643 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:55.179530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:55.179643 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.179529    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wmbds" podUID="b322eaf2-56c8-448e-83d0-c4fb746fb0a8"
Jul 27 20:51:55.179643 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.179625    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t8lbq" podUID="ab76bb44-d4bb-467c-9c4b-275f4f60b303"
Jul 27 20:51:55.179846 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.179712    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4br5p" podUID="59a571a8-5d4f-4cb9-8010-f2257b46dbd3"
Jul 27 20:51:55.783441 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:55.783400    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:55.783966 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.783578    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:55.783966 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.783646    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret podName:59a571a8-5d4f-4cb9-8010-f2257b46dbd3 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:27.783627714 +0000 UTC m=+65.149193540 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret") pod "global-pull-secret-syncer-4br5p" (UID: "59a571a8-5d4f-4cb9-8010-f2257b46dbd3") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:55.884020 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:55.883979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:55.884227 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:55.884067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:55.884227 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.884153    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:55.884227 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.884179    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:55.884227 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.884211    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:55.884227 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.884223    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4cm6 for pod openshift-network-diagnostics/network-check-target-t8lbq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:55.884500 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.884237    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs podName:b322eaf2-56c8-448e-83d0-c4fb746fb0a8 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:27.88421585 +0000 UTC m=+65.249781680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs") pod "network-metrics-daemon-wmbds" (UID: "b322eaf2-56c8-448e-83d0-c4fb746fb0a8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:55.884500 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:55.884263    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6 podName:ab76bb44-d4bb-467c-9c4b-275f4f60b303 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:27.884252404 +0000 UTC m=+65.249818229 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4cm6" (UniqueName: "kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6") pod "network-check-target-t8lbq" (UID: "ab76bb44-d4bb-467c-9c4b-275f4f60b303") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:56.086516 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:56.086428    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:56.187965 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:56.187924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:56.932244 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:56.932220    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-91.ec2.internal" event="NodeReady"
Jul 27 20:51:56.932662 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:56.932349    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 20:51:56.993069 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:56.993042    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-48trx"]
Jul 27 20:51:57.007608 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.007582    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hml2p"]
Jul 27 20:51:57.007731 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.007714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.010168 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.010142    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 20:51:57.010347 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.010301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 20:51:57.010347 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.010334    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2qj4g\""
Jul 27 20:51:57.025849 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.025801    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-48trx"]
Jul 27 20:51:57.025989 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.025877    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hml2p"]
Jul 27 20:51:57.026057 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.026015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hml2p"
Jul 27 20:51:57.028670 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.028648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 20:51:57.028768 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.028723    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 20:51:57.028768 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.028725    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cgmf5\""
Jul 27 20:51:57.028865 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.028725    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 20:51:57.093537 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.093504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f7262a1-1f6a-4252-a2dc-99b96a2422c2-cert\") pod \"ingress-canary-hml2p\" (UID: \"6f7262a1-1f6a-4252-a2dc-99b96a2422c2\") " pod="openshift-ingress-canary/ingress-canary-hml2p"
Jul 27 20:51:57.093720 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.093565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjw6q\" (UniqueName: \"kubernetes.io/projected/6f7262a1-1f6a-4252-a2dc-99b96a2422c2-kube-api-access-pjw6q\") pod \"ingress-canary-hml2p\" (UID: \"6f7262a1-1f6a-4252-a2dc-99b96a2422c2\") " pod="openshift-ingress-canary/ingress-canary-hml2p"
Jul 27 20:51:57.093720 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.093588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f5a733-8e79-4afd-8bf1-3b8ced670805-tmp-dir\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.093720 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.093646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.093720 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.093673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cctfd\" (UniqueName: \"kubernetes.io/projected/f2f5a733-8e79-4afd-8bf1-3b8ced670805-kube-api-access-cctfd\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.093720 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.093709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f5a733-8e79-4afd-8bf1-3b8ced670805-config-volume\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.178900 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.178867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:51:57.179035 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.178908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:51:57.179035 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.178867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:51:57.183722 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.183695    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2w5wq\""
Jul 27 20:51:57.183885 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.183748    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:51:57.183885 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.183769    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:51:57.183885 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.183780    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6wczd\""
Jul 27 20:51:57.184116 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.184089    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:51:57.184116 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.184095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:51:57.194035 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.194014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f7262a1-1f6a-4252-a2dc-99b96a2422c2-cert\") pod \"ingress-canary-hml2p\" (UID: \"6f7262a1-1f6a-4252-a2dc-99b96a2422c2\") " pod="openshift-ingress-canary/ingress-canary-hml2p"
Jul 27 20:51:57.194117 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.194077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjw6q\" (UniqueName: \"kubernetes.io/projected/6f7262a1-1f6a-4252-a2dc-99b96a2422c2-kube-api-access-pjw6q\") pod \"ingress-canary-hml2p\" (UID: \"6f7262a1-1f6a-4252-a2dc-99b96a2422c2\") " pod="openshift-ingress-canary/ingress-canary-hml2p"
Jul 27 20:51:57.194117 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.194104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f5a733-8e79-4afd-8bf1-3b8ced670805-tmp-dir\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.194228 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.194159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.194228 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.194202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cctfd\" (UniqueName: \"kubernetes.io/projected/f2f5a733-8e79-4afd-8bf1-3b8ced670805-kube-api-access-cctfd\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.194325 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.194241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f5a733-8e79-4afd-8bf1-3b8ced670805-config-volume\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.194325 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:57.194277    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:51:57.194418 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:57.194338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls podName:f2f5a733-8e79-4afd-8bf1-3b8ced670805 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:57.694318311 +0000 UTC m=+35.059884148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls") pod "dns-default-48trx" (UID: "f2f5a733-8e79-4afd-8bf1-3b8ced670805") : secret "dns-default-metrics-tls" not found
Jul 27 20:51:57.194469 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.194437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f5a733-8e79-4afd-8bf1-3b8ced670805-tmp-dir\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.194820 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.194801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f5a733-8e79-4afd-8bf1-3b8ced670805-config-volume\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.198557 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.198538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f7262a1-1f6a-4252-a2dc-99b96a2422c2-cert\") pod \"ingress-canary-hml2p\" (UID: \"6f7262a1-1f6a-4252-a2dc-99b96a2422c2\") " pod="openshift-ingress-canary/ingress-canary-hml2p"
Jul 27 20:51:57.203640 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.203616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjw6q\" (UniqueName: \"kubernetes.io/projected/6f7262a1-1f6a-4252-a2dc-99b96a2422c2-kube-api-access-pjw6q\") pod \"ingress-canary-hml2p\" (UID: \"6f7262a1-1f6a-4252-a2dc-99b96a2422c2\") " pod="openshift-ingress-canary/ingress-canary-hml2p"
Jul 27 20:51:57.204518 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.204498    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cctfd\" (UniqueName: \"kubernetes.io/projected/f2f5a733-8e79-4afd-8bf1-3b8ced670805-kube-api-access-cctfd\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.355680 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.355652    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hml2p"
Jul 27 20:51:57.438332 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.438305    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff" containerID="024c774ea8f19f8a18dc3908fa30bd786018ca8bcbd0d00d0dd27319d38b1826" exitCode=0
Jul 27 20:51:57.438483 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.438340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" event={"ID":"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff","Type":"ContainerDied","Data":"024c774ea8f19f8a18dc3908fa30bd786018ca8bcbd0d00d0dd27319d38b1826"}
Jul 27 20:51:57.546629 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.546484    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hml2p"]
Jul 27 20:51:57.549516 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:51:57.549491    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7262a1_1f6a_4252_a2dc_99b96a2422c2.slice/crio-d7a43d2171a737cfbaa588d369c0b31541b9d68d83bef88f625f04c51187fd2f WatchSource:0}: Error finding container d7a43d2171a737cfbaa588d369c0b31541b9d68d83bef88f625f04c51187fd2f: Status 404 returned error can't find the container with id d7a43d2171a737cfbaa588d369c0b31541b9d68d83bef88f625f04c51187fd2f
Jul 27 20:51:57.697932 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:57.697842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:57.698049 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:57.697952    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:51:57.698049 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:57.697999    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls podName:f2f5a733-8e79-4afd-8bf1-3b8ced670805 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:58.697986419 +0000 UTC m=+36.063552245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls") pod "dns-default-48trx" (UID: "f2f5a733-8e79-4afd-8bf1-3b8ced670805") : secret "dns-default-metrics-tls" not found
Jul 27 20:51:58.444784 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:58.444748    2577 generic.go:356] "Generic (PLEG): container finished" podID="af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff" containerID="ced59807cbf4bf1a9d4410318c43629a8b731d1924edfaed64e5f6afd7719a92" exitCode=0
Jul 27 20:51:58.445450 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:58.444842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" event={"ID":"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff","Type":"ContainerDied","Data":"ced59807cbf4bf1a9d4410318c43629a8b731d1924edfaed64e5f6afd7719a92"}
Jul 27 20:51:58.446240 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:58.446216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hml2p" event={"ID":"6f7262a1-1f6a-4252-a2dc-99b96a2422c2","Type":"ContainerStarted","Data":"d7a43d2171a737cfbaa588d369c0b31541b9d68d83bef88f625f04c51187fd2f"}
Jul 27 20:51:58.705071 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:58.704988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:51:58.705246 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:58.705150    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:51:58.705313 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:51:58.705235    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls podName:f2f5a733-8e79-4afd-8bf1-3b8ced670805 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:00.705216473 +0000 UTC m=+38.070782321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls") pod "dns-default-48trx" (UID: "f2f5a733-8e79-4afd-8bf1-3b8ced670805") : secret "dns-default-metrics-tls" not found
Jul 27 20:51:59.447517 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:59.447475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:59.455589 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:59.455553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" event={"ID":"af3fea87-92d3-4b71-b2a0-edf0b0b1f9ff","Type":"ContainerStarted","Data":"aee5a484f4ef630718352e17165f3d0ecb761edc6c1741bcac015bdb5d60711e"}
Jul 27 20:51:59.483803 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:59.483750    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jh5v7" podStartSLOduration=6.950978704 podStartE2EDuration="36.483730848s" totalImagesPullingTime="29.532752144s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:59.483218141 +0000 UTC m=+36.848783993" watchObservedRunningTime="2026-07-27 20:51:59.483730848 +0000 UTC m=+36.849296697"
Jul 27 20:51:59.535135 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:59.535099    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:59.625284 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:59.625243    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:51:59.727722 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:51:59.727656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:00.130603 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.130580    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g7ml4"]
Jul 27 20:52:00.146257 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.146240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.149394 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.149376    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 20:52:00.149789 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.149775    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 20:52:00.150704 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.150672    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qwd8q\""
Jul 27 20:52:00.150819 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.150672    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 20:52:00.150888 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.150670    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 20:52:00.152105 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.152085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g7ml4"]
Jul 27 20:52:00.158022 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.157997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:00.215597 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.215575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-data-volume\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.215707 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.215624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-crio-socket\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.215707 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.215658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.215707 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.215684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kh6q\" (UniqueName: \"kubernetes.io/projected/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-api-access-7kh6q\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.215833 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.215737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.226053 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.226017    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:00.316148 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.316111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-data-volume\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.316332 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.316177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-crio-socket\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.316332 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.316230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.316332 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.316258    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kh6q\" (UniqueName: \"kubernetes.io/projected/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-api-access-7kh6q\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.316483 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.316371    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:00.316483 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.316446    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls podName:54e17ea4-433d-4bac-8a82-50ca06d6c5a0 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:00.816425115 +0000 UTC m=+38.181990944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7ml4" (UID: "54e17ea4-433d-4bac-8a82-50ca06d6c5a0") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:00.316483 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.316446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-crio-socket\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.316483 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.316478    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-data-volume\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.316646 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.316494    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.316646 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.316549    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:00.316646 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.316612    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm podName:54e17ea4-433d-4bac-8a82-50ca06d6c5a0 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:00.816600234 +0000 UTC m=+38.182166060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g7ml4" (UID: "54e17ea4-433d-4bac-8a82-50ca06d6c5a0") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:00.330035 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.330008    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:00.333159 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.333054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kh6q\" (UniqueName: \"kubernetes.io/projected/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-api-access-7kh6q\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.354230 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.354199    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k"]
Jul 27 20:52:00.363168 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.363114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k"
Jul 27 20:52:00.366485 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.366463    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xtxln\""
Jul 27 20:52:00.366602 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.366466    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 27 20:52:00.366602 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.366533    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 27 20:52:00.367950 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.367928    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k"]
Jul 27 20:52:00.427408 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.427369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:00.459224 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.459166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hml2p" event={"ID":"6f7262a1-1f6a-4252-a2dc-99b96a2422c2","Type":"ContainerStarted","Data":"44ea60cd9f0c7edfbb8b64f40f009d7d858437a64a179991e717569fa812e020"}
Jul 27 20:52:00.475498 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.475442    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hml2p" podStartSLOduration=1.920248032 podStartE2EDuration="4.475420042s" totalImagesPullingTime="2.55517201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:00.475402394 +0000 UTC m=+37.840968234" watchObservedRunningTime="2026-07-27 20:52:00.475420042 +0000 UTC m=+37.840985889"
Jul 27 20:52:00.518206 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.518162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rsx\" (UniqueName: \"kubernetes.io/projected/236b13d5-7d68-4494-8f5d-108b30e2fe9e-kube-api-access-m7rsx\") pod \"migrator-6865c9967-dn44k\" (UID: \"236b13d5-7d68-4494-8f5d-108b30e2fe9e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k"
Jul 27 20:52:00.519128 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.519090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:00.619597 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.619489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsx\" (UniqueName: \"kubernetes.io/projected/236b13d5-7d68-4494-8f5d-108b30e2fe9e-kube-api-access-m7rsx\") pod \"migrator-6865c9967-dn44k\" (UID: \"236b13d5-7d68-4494-8f5d-108b30e2fe9e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k"
Jul 27 20:52:00.628259 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.628236    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rsx\" (UniqueName: \"kubernetes.io/projected/236b13d5-7d68-4494-8f5d-108b30e2fe9e-kube-api-access-m7rsx\") pod \"migrator-6865c9967-dn44k\" (UID: \"236b13d5-7d68-4494-8f5d-108b30e2fe9e\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k"
Jul 27 20:52:00.673436 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.673399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k"
Jul 27 20:52:00.683433 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.683409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:00.720392 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.720361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:52:00.720552 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.720493    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:00.720597 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.720560    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls podName:f2f5a733-8e79-4afd-8bf1-3b8ced670805 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:04.720537443 +0000 UTC m=+42.086103268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls") pod "dns-default-48trx" (UID: "f2f5a733-8e79-4afd-8bf1-3b8ced670805") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:00.785128 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.785098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:00.810209 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.810158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k"]
Jul 27 20:52:00.814751 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:00.814727    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236b13d5_7d68_4494_8f5d_108b30e2fe9e.slice/crio-72adcb2383799c921172b8d5195eebb6cc6e444d2329ab75a1e077e2137b9be3 WatchSource:0}: Error finding container 72adcb2383799c921172b8d5195eebb6cc6e444d2329ab75a1e077e2137b9be3: Status 404 returned error can't find the container with id 72adcb2383799c921172b8d5195eebb6cc6e444d2329ab75a1e077e2137b9be3
Jul 27 20:52:00.821400 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.821382    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.821474 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.821429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:00.821535 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.821518    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:00.821573 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.821545    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:00.821609 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.821577    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls podName:54e17ea4-433d-4bac-8a82-50ca06d6c5a0 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:01.821559129 +0000 UTC m=+39.187124954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7ml4" (UID: "54e17ea4-433d-4bac-8a82-50ca06d6c5a0") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:00.821609 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:00.821607    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm podName:54e17ea4-433d-4bac-8a82-50ca06d6c5a0 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:01.821596223 +0000 UTC m=+39.187162048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g7ml4" (UID: "54e17ea4-433d-4bac-8a82-50ca06d6c5a0") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:00.959026 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:00.959001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:01.060231 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:01.060175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:01.465889 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:01.465851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k" event={"ID":"236b13d5-7d68-4494-8f5d-108b30e2fe9e","Type":"ContainerStarted","Data":"72adcb2383799c921172b8d5195eebb6cc6e444d2329ab75a1e077e2137b9be3"}
Jul 27 20:52:01.830130 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:01.830087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:01.830334 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:01.830159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:01.830334 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:01.830262    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:01.830334 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:01.830301    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:01.830334 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:01.830329    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls podName:54e17ea4-433d-4bac-8a82-50ca06d6c5a0 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:03.830310732 +0000 UTC m=+41.195876560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7ml4" (UID: "54e17ea4-433d-4bac-8a82-50ca06d6c5a0") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:01.830486 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:01.830355    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm podName:54e17ea4-433d-4bac-8a82-50ca06d6c5a0 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:03.83034239 +0000 UTC m=+41.195908215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g7ml4" (UID: "54e17ea4-433d-4bac-8a82-50ca06d6c5a0") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:02.122418 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.122320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:02.303482 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.303456    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x5xkb_214751b9-5778-4f01-8602-5b89512a8384/dns-node-resolver/0.log"
Jul 27 20:52:02.580568 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.580417    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-57zh5"]
Jul 27 20:52:02.587346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.587323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:02.595831 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.595811    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-57zh5"]
Jul 27 20:52:02.595952 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.595933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.598715 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.598697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 20:52:02.598805 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.598789    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 20:52:02.599781 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.599760    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 20:52:02.599781 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.599762    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 20:52:02.599954 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.599784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7q5s5\""
Jul 27 20:52:02.683749 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.683726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:02.737304 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.737275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7934d9d6-3314-4099-845a-00132372809a-signing-cabundle\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.737453 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.737312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7934d9d6-3314-4099-845a-00132372809a-signing-key\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.737453 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.737383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs2nk\" (UniqueName: \"kubernetes.io/projected/7934d9d6-3314-4099-845a-00132372809a-kube-api-access-qs2nk\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.837958 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.837871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7934d9d6-3314-4099-845a-00132372809a-signing-cabundle\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.837958 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.837920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7934d9d6-3314-4099-845a-00132372809a-signing-key\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.838147 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.838123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs2nk\" (UniqueName: \"kubernetes.io/projected/7934d9d6-3314-4099-845a-00132372809a-kube-api-access-qs2nk\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.838681 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.838614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7934d9d6-3314-4099-845a-00132372809a-signing-cabundle\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.840698 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.840674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7934d9d6-3314-4099-845a-00132372809a-signing-key\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.846371 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.846350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs2nk\" (UniqueName: \"kubernetes.io/projected/7934d9d6-3314-4099-845a-00132372809a-kube-api-access-qs2nk\") pod \"service-ca-658d7b7c78-57zh5\" (UID: \"7934d9d6-3314-4099-845a-00132372809a\") " pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.905490 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.905450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-57zh5"
Jul 27 20:52:02.909624 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:02.909587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:03.049467 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.049436    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-57zh5"]
Jul 27 20:52:03.052749 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:03.052722    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7934d9d6_3314_4099_845a_00132372809a.slice/crio-29f842e90ac08bd414fbc57eb8c81192451ceffbadb42280ec268178a99879e1 WatchSource:0}: Error finding container 29f842e90ac08bd414fbc57eb8c81192451ceffbadb42280ec268178a99879e1: Status 404 returned error can't find the container with id 29f842e90ac08bd414fbc57eb8c81192451ceffbadb42280ec268178a99879e1
Jul 27 20:52:03.313559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.313534    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ph9gv_739950fc-859d-4145-b60c-b5dc0811d195/node-ca/0.log"
Jul 27 20:52:03.472047 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.472017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k" event={"ID":"236b13d5-7d68-4494-8f5d-108b30e2fe9e","Type":"ContainerStarted","Data":"b331af130b7cc151443ecc5ec9078fd9d5e6514d7977c8e7cbfd21041f8cb9e4"}
Jul 27 20:52:03.472179 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.472070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k" event={"ID":"236b13d5-7d68-4494-8f5d-108b30e2fe9e","Type":"ContainerStarted","Data":"dba88eb2ea2914b82ca28e38e4af3d696d7dee4cdf17d4751416bb8eb08a4fae"}
Jul 27 20:52:03.473080 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.473060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-57zh5" event={"ID":"7934d9d6-3314-4099-845a-00132372809a","Type":"ContainerStarted","Data":"29f842e90ac08bd414fbc57eb8c81192451ceffbadb42280ec268178a99879e1"}
Jul 27 20:52:03.490404 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.490350    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dn44k" podStartSLOduration=1.797027383 podStartE2EDuration="3.490334795s" totalImagesPullingTime="1.693307412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:03.489374824 +0000 UTC m=+40.854940672" watchObservedRunningTime="2026-07-27 20:52:03.490334795 +0000 UTC m=+40.855900642"
Jul 27 20:52:03.846339 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.846302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:03.846805 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.846378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:03.846805 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:03.846490    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:03.846805 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:03.846578    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls podName:54e17ea4-433d-4bac-8a82-50ca06d6c5a0 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:07.846555864 +0000 UTC m=+45.212121690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7ml4" (UID: "54e17ea4-433d-4bac-8a82-50ca06d6c5a0") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:03.846961 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:03.846930    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:04.104488 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:04.104404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hml2p_6f7262a1-1f6a-4252-a2dc-99b96a2422c2/serve-healthcheck-canary/0.log"
Jul 27 20:52:04.190523 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:04.190450    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:04.755816 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:04.755778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:52:04.756011 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:04.755966    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:04.756072 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:04.756040    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls podName:f2f5a733-8e79-4afd-8bf1-3b8ced670805 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:12.75602324 +0000 UTC m=+50.121589085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls") pod "dns-default-48trx" (UID: "f2f5a733-8e79-4afd-8bf1-3b8ced670805") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:04.803432 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:04.803355    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:04.903456 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:04.903419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:06.483276 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:06.483239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-57zh5" event={"ID":"7934d9d6-3314-4099-845a-00132372809a","Type":"ContainerStarted","Data":"5becabe478803d4e5696e4d84d8dd8993434343e65ee75e16b3a8affcfab9e3f"}
Jul 27 20:52:06.504135 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:06.504075    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-57zh5" podStartSLOduration=2.1343677899999998 podStartE2EDuration="4.504039599s" totalImagesPullingTime="2.369671809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:06.501917315 +0000 UTC m=+43.867483162" watchObservedRunningTime="2026-07-27 20:52:06.504039599 +0000 UTC m=+43.869605438"
Jul 27 20:52:06.576590 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:06.576511    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:06.858460 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:06.858369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:06.944386 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:06.944349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:07.066908 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:07.066873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:07.200130 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:07.200037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:07.313201 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:07.313157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:07.723374 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:07.723323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:07.881871 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:07.881835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:07.882039 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:07.881994    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:07.882080 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:07.882060    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls podName:54e17ea4-433d-4bac-8a82-50ca06d6c5a0 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:15.882041863 +0000 UTC m=+53.247607687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g7ml4" (UID: "54e17ea4-433d-4bac-8a82-50ca06d6c5a0") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:08.490349 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:08.490312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:08.668542 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:08.668496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:08.868700 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:08.868628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:09.077991 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:09.077956    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:09.270724 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:09.270691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:10.186425 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:10.186393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:10.291787 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:10.291756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:10.477365 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:10.477280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:10.742552 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:10.742487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:10.869956 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:10.869929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:11.070311 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:11.070234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:11.139250 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:11.139207    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:11.276202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:11.276162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:12.502862 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:12.502830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:12.674728 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:12.674691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:12.821003 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:12.820925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:52:12.823494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:12.823468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2f5a733-8e79-4afd-8bf1-3b8ced670805-metrics-tls\") pod \"dns-default-48trx\" (UID: \"f2f5a733-8e79-4afd-8bf1-3b8ced670805\") " pod="openshift-dns/dns-default-48trx"
Jul 27 20:52:12.865566 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:12.865534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:12.932437 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:12.932410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:12.951554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:12.951526    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-48trx"
Jul 27 20:52:13.067541 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:13.067508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:13.100244 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:13.100146    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-48trx"]
Jul 27 20:52:13.104326 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:13.104295    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f5a733_8e79_4afd_8bf1_3b8ced670805.slice/crio-2c4979e23304e8a52f02750854924e3cae7d827856a16a53b3cdb3af65672159 WatchSource:0}: Error finding container 2c4979e23304e8a52f02750854924e3cae7d827856a16a53b3cdb3af65672159: Status 404 returned error can't find the container with id 2c4979e23304e8a52f02750854924e3cae7d827856a16a53b3cdb3af65672159
Jul 27 20:52:13.269740 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:13.269689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:13.500986 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:13.500955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48trx" event={"ID":"f2f5a733-8e79-4afd-8bf1-3b8ced670805","Type":"ContainerStarted","Data":"2c4979e23304e8a52f02750854924e3cae7d827856a16a53b3cdb3af65672159"}
Jul 27 20:52:14.279751 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:14.279716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:14.478808 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:14.478762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:14.889910 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:14.889880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:15.073661 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:15.073625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:15.482782 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:15.482749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:15.510170 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:15.510139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48trx" event={"ID":"f2f5a733-8e79-4afd-8bf1-3b8ced670805","Type":"ContainerStarted","Data":"7ae7146e704cec9973f0882c720779470643b67a6c707f7407f1956eebda8f5a"}
Jul 27 20:52:15.587421 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:15.587390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:15.948566 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:15.948521    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:15.950927 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:15.950899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/54e17ea4-433d-4bac-8a82-50ca06d6c5a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g7ml4\" (UID: \"54e17ea4-433d-4bac-8a82-50ca06d6c5a0\") " pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:16.068928 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.068900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:16.072804 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.072786    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g7ml4"
Jul 27 20:52:16.199226 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.199136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g7ml4"]
Jul 27 20:52:16.202169 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:16.202146    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e17ea4_433d_4bac_8a82_50ca06d6c5a0.slice/crio-19f127a0a37182f93c9f16d8911c2d2da73232f4f878186141858da55e3e3910 WatchSource:0}: Error finding container 19f127a0a37182f93c9f16d8911c2d2da73232f4f878186141858da55e3e3910: Status 404 returned error can't find the container with id 19f127a0a37182f93c9f16d8911c2d2da73232f4f878186141858da55e3e3910
Jul 27 20:52:16.514740 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.514705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-48trx" event={"ID":"f2f5a733-8e79-4afd-8bf1-3b8ced670805","Type":"ContainerStarted","Data":"1a1bf9ae50d56ce89cb63f6bd2ed2b22377db49ed6cdc47eb3c90eb10d4b9b56"}
Jul 27 20:52:16.515160 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.514791    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-48trx"
Jul 27 20:52:16.515903 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.515882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g7ml4" event={"ID":"54e17ea4-433d-4bac-8a82-50ca06d6c5a0","Type":"ContainerStarted","Data":"19f127a0a37182f93c9f16d8911c2d2da73232f4f878186141858da55e3e3910"}
Jul 27 20:52:16.534643 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.534596    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-48trx" podStartSLOduration=18.253999766 podStartE2EDuration="20.534582303s" totalImagesPullingTime="2.280582537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:16.533076504 +0000 UTC m=+53.898642351" watchObservedRunningTime="2026-07-27 20:52:16.534582303 +0000 UTC m=+53.900148149"
Jul 27 20:52:16.665420 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.665383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:16.868558 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:16.868487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:17.074457 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:17.074390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:17.343630 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:17.343602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:17.431061 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:17.431021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:17.601316 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:17.601151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:17.988629 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:17.988593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:18.074436 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:18.074392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:18.182142 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:18.182105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:18.282420 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:18.282332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:18.386771 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:18.386739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:18.492936 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:18.492890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.005305 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:19.005274    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.204815 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:19.204786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.277665 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:19.277584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.806780 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:19.806728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:19.878561 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:19.878505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:20.016329 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:20.016283    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:20.082890 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:20.082808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:21.533451 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:21.533417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g7ml4" event={"ID":"54e17ea4-433d-4bac-8a82-50ca06d6c5a0","Type":"ContainerStarted","Data":"a9c79949d1932d7e8d3238bfcbbb2125aef80c6b9285bb991b31182e14787ac9"}
Jul 27 20:52:23.438987 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:23.438957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ztzwl"
Jul 27 20:52:23.977926 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:23.977884    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"]
Jul 27 20:52:23.999707 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:23.999677    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"]
Jul 27 20:52:23.999876 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:23.999848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"
Jul 27 20:52:24.003057 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.003030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 20:52:24.003271 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.003030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 20:52:24.003979 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.003958    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 20:52:24.004270 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.004250    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 20:52:24.004519 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.004497    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-w5htw\""
Jul 27 20:52:24.004622 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.004603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"]
Jul 27 20:52:24.004712 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.004698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.007254 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.007232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 20:52:24.026469 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.026446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"]
Jul 27 20:52:24.121355 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.121323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-tmp\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.121504 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.121362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afaff80e-c3cc-4fa5-a12a-433e23cc9e2e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d7b4f96-5pd9h\" (UID: \"afaff80e-c3cc-4fa5-a12a-433e23cc9e2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"
Jul 27 20:52:24.121504 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.121417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.121504 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.121447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsk6x\" (UniqueName: \"kubernetes.io/projected/afaff80e-c3cc-4fa5-a12a-433e23cc9e2e-kube-api-access-qsk6x\") pod \"managed-serviceaccount-addon-agent-54d7b4f96-5pd9h\" (UID: \"afaff80e-c3cc-4fa5-a12a-433e23cc9e2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"
Jul 27 20:52:24.121622 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.121497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-728gp\" (UniqueName: \"kubernetes.io/projected/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-kube-api-access-728gp\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.149992 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.149965    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:24.221905 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.221859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afaff80e-c3cc-4fa5-a12a-433e23cc9e2e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d7b4f96-5pd9h\" (UID: \"afaff80e-c3cc-4fa5-a12a-433e23cc9e2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"
Jul 27 20:52:24.222061 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.221912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.222061 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.221932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsk6x\" (UniqueName: \"kubernetes.io/projected/afaff80e-c3cc-4fa5-a12a-433e23cc9e2e-kube-api-access-qsk6x\") pod \"managed-serviceaccount-addon-agent-54d7b4f96-5pd9h\" (UID: \"afaff80e-c3cc-4fa5-a12a-433e23cc9e2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"
Jul 27 20:52:24.222061 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.221958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-728gp\" (UniqueName: \"kubernetes.io/projected/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-kube-api-access-728gp\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.222061 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.222042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-tmp\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.222454 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.222432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-tmp\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.224636 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.224609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afaff80e-c3cc-4fa5-a12a-433e23cc9e2e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54d7b4f96-5pd9h\" (UID: \"afaff80e-c3cc-4fa5-a12a-433e23cc9e2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"
Jul 27 20:52:24.224773 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.224750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.234352 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.234332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsk6x\" (UniqueName: \"kubernetes.io/projected/afaff80e-c3cc-4fa5-a12a-433e23cc9e2e-kube-api-access-qsk6x\") pod \"managed-serviceaccount-addon-agent-54d7b4f96-5pd9h\" (UID: \"afaff80e-c3cc-4fa5-a12a-433e23cc9e2e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"
Jul 27 20:52:24.235967 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.235946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-728gp\" (UniqueName: \"kubernetes.io/projected/3c8c6773-549b-45e2-8dd5-3eb3441aa67a-kube-api-access-728gp\") pod \"klusterlet-addon-workmgr-74d8ddb6bf-96z2l\" (UID: \"3c8c6773-549b-45e2-8dd5-3eb3441aa67a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.278033 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.277988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:24.318383 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.318348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"
Jul 27 20:52:24.325910 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.325891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:24.410977 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.410943    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:24.505419 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:24.505339    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:25.006401 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:25.006376    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h"]
Jul 27 20:52:25.009286 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:25.009257    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafaff80e_c3cc_4fa5_a12a_433e23cc9e2e.slice/crio-67680c7cfff4f4f43c28f6babf51f21d0fd12e58693aa3d3cccd4235bf9e302d WatchSource:0}: Error finding container 67680c7cfff4f4f43c28f6babf51f21d0fd12e58693aa3d3cccd4235bf9e302d: Status 404 returned error can't find the container with id 67680c7cfff4f4f43c28f6babf51f21d0fd12e58693aa3d3cccd4235bf9e302d
Jul 27 20:52:25.023141 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:25.023119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"]
Jul 27 20:52:25.025161 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:25.025129    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8c6773_549b_45e2_8dd5_3eb3441aa67a.slice/crio-47805bd3262a3a1c16c626e5804fdc688c1e9e6a497e5ca6e328157c0d72f884 WatchSource:0}: Error finding container 47805bd3262a3a1c16c626e5804fdc688c1e9e6a497e5ca6e328157c0d72f884: Status 404 returned error can't find the container with id 47805bd3262a3a1c16c626e5804fdc688c1e9e6a497e5ca6e328157c0d72f884
Jul 27 20:52:25.433252 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:25.433208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:25.549553 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:25.549518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g7ml4" event={"ID":"54e17ea4-433d-4bac-8a82-50ca06d6c5a0","Type":"ContainerStarted","Data":"c207fde2b40dca5a1b54f0c8632031953a62952766380d8c5f6e3d860b1a6f0d"}
Jul 27 20:52:25.550814 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:25.550783    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h" event={"ID":"afaff80e-c3cc-4fa5-a12a-433e23cc9e2e","Type":"ContainerStarted","Data":"67680c7cfff4f4f43c28f6babf51f21d0fd12e58693aa3d3cccd4235bf9e302d"}
Jul 27 20:52:25.551969 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:25.551941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l" event={"ID":"3c8c6773-549b-45e2-8dd5-3eb3441aa67a","Type":"ContainerStarted","Data":"47805bd3262a3a1c16c626e5804fdc688c1e9e6a497e5ca6e328157c0d72f884"}
Jul 27 20:52:25.838737 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:25.838700    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:25.928825 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:25.928754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:26.522163 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:26.521291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-48trx"
Jul 27 20:52:26.665379 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:26.665343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:26.782433 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:26.782298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:27.854196 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.854155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:52:27.857315 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.857294    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:52:27.867488 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.867459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/59a571a8-5d4f-4cb9-8010-f2257b46dbd3-original-pull-secret\") pod \"global-pull-secret-syncer-4br5p\" (UID: \"59a571a8-5d4f-4cb9-8010-f2257b46dbd3\") " pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:52:27.954562 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.954527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:52:27.954737 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.954624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:52:27.958060 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.958034    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:52:27.958217 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.958072    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:52:27.968416 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.968386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b322eaf2-56c8-448e-83d0-c4fb746fb0a8-metrics-certs\") pod \"network-metrics-daemon-wmbds\" (UID: \"b322eaf2-56c8-448e-83d0-c4fb746fb0a8\") " pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:52:27.968666 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.968645    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:52:27.978803 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:27.978772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cm6\" (UniqueName: \"kubernetes.io/projected/ab76bb44-d4bb-467c-9c4b-275f4f60b303-kube-api-access-h4cm6\") pod \"network-check-target-t8lbq\" (UID: \"ab76bb44-d4bb-467c-9c4b-275f4f60b303\") " pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:52:28.088562 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:28.088522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4br5p"
Jul 27 20:52:28.097454 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:28.097428    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2w5wq\""
Jul 27 20:52:28.101051 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:28.101027    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6wczd\""
Jul 27 20:52:28.104515 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:28.104435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wmbds"
Jul 27 20:52:28.108858 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:28.108841    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:52:30.231427 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.231393    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-786676f497-jqk9j"]
Jul 27 20:52:30.233530 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.233510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.237385 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.237361    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 20:52:30.238224 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.238206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 20:52:30.238344 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.238310    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j8rtq\""
Jul 27 20:52:30.238416 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.238309    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 20:52:30.238498 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.238471    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 20:52:30.238498 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.238480    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 20:52:30.238605 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.238528    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 20:52:30.238703 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.238607    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 20:52:30.248682 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.248663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786676f497-jqk9j"]
Jul 27 20:52:30.269328 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.269298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-oauth-config\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.269420 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.269350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-service-ca\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.269420 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.269387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-serving-cert\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.269420 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.269412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-console-config\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.269580 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.269435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq64v\" (UniqueName: \"kubernetes.io/projected/542fcae9-4569-43c6-8f6b-002e481dda1b-kube-api-access-bq64v\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.269580 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.269551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-oauth-serving-cert\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.370792 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.370750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-oauth-serving-cert\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.370990 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.370816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-oauth-config\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.370990 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.370859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-service-ca\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.370990 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.370890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-serving-cert\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.370990 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.370913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-console-config\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.370990 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.370938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq64v\" (UniqueName: \"kubernetes.io/projected/542fcae9-4569-43c6-8f6b-002e481dda1b-kube-api-access-bq64v\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.371594 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.371572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-oauth-serving-cert\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.371698 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.371667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-service-ca\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.371780 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.371765    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-console-config\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.374038 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.374011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-oauth-config\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.374142 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.374055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-serving-cert\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.381585 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.381561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq64v\" (UniqueName: \"kubernetes.io/projected/542fcae9-4569-43c6-8f6b-002e481dda1b-kube-api-access-bq64v\") pod \"console-786676f497-jqk9j\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") " pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.545174 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.545082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:30.579179 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.579148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:30.644055 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:30.644006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:31.336609 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.336575    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t8lbq"]
Jul 27 20:52:31.353323 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.353295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4br5p"]
Jul 27 20:52:31.357557 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:31.357519    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a571a8_5d4f_4cb9_8010_f2257b46dbd3.slice/crio-1f5ce0bd1dea9e3a54cd8dcf26849108d8d54e5d5efda6d7bfaacef6d3f4aeb9 WatchSource:0}: Error finding container 1f5ce0bd1dea9e3a54cd8dcf26849108d8d54e5d5efda6d7bfaacef6d3f4aeb9: Status 404 returned error can't find the container with id 1f5ce0bd1dea9e3a54cd8dcf26849108d8d54e5d5efda6d7bfaacef6d3f4aeb9
Jul 27 20:52:31.372100 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.371945    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wmbds"]
Jul 27 20:52:31.375590 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:31.375561    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb322eaf2_56c8_448e_83d0_c4fb746fb0a8.slice/crio-1cbe04f4278a797cd5196e5a0d834ec8c667d30997eebb84588f51dfa7e072c9 WatchSource:0}: Error finding container 1cbe04f4278a797cd5196e5a0d834ec8c667d30997eebb84588f51dfa7e072c9: Status 404 returned error can't find the container with id 1cbe04f4278a797cd5196e5a0d834ec8c667d30997eebb84588f51dfa7e072c9
Jul 27 20:52:31.387034 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.387008    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786676f497-jqk9j"]
Jul 27 20:52:31.393039 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:31.392965    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542fcae9_4569_43c6_8f6b_002e481dda1b.slice/crio-a1bebd9bfa08398d029e768a7ea9641e556f71f6e4a067ab0f5d9007b6d088ec WatchSource:0}: Error finding container a1bebd9bfa08398d029e768a7ea9641e556f71f6e4a067ab0f5d9007b6d088ec: Status 404 returned error can't find the container with id a1bebd9bfa08398d029e768a7ea9641e556f71f6e4a067ab0f5d9007b6d088ec
Jul 27 20:52:31.579219 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.579164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8lbq" event={"ID":"ab76bb44-d4bb-467c-9c4b-275f4f60b303","Type":"ContainerStarted","Data":"e36a5788ef787b2c09c5f6c593ba50b14d1faa9b9d07dd3340a0b4a2727028f3"}
Jul 27 20:52:31.580409 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.580376    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmbds" event={"ID":"b322eaf2-56c8-448e-83d0-c4fb746fb0a8","Type":"ContainerStarted","Data":"1cbe04f4278a797cd5196e5a0d834ec8c667d30997eebb84588f51dfa7e072c9"}
Jul 27 20:52:31.581633 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.581608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786676f497-jqk9j" event={"ID":"542fcae9-4569-43c6-8f6b-002e481dda1b","Type":"ContainerStarted","Data":"a1bebd9bfa08398d029e768a7ea9641e556f71f6e4a067ab0f5d9007b6d088ec"}
Jul 27 20:52:31.583796 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.583765    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g7ml4" event={"ID":"54e17ea4-433d-4bac-8a82-50ca06d6c5a0","Type":"ContainerStarted","Data":"342b5db7c130d75b87281f8f72a4fb64afc0ac69761ca397baed051bae31b3c6"}
Jul 27 20:52:31.585554 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.585516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h" event={"ID":"afaff80e-c3cc-4fa5-a12a-433e23cc9e2e","Type":"ContainerStarted","Data":"6c4bcabf18ab7b9158232b34baf2b922b0c9710cb5615d5402b393c8cd6cb30d"}
Jul 27 20:52:31.586958 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.586931    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4br5p" event={"ID":"59a571a8-5d4f-4cb9-8010-f2257b46dbd3","Type":"ContainerStarted","Data":"1f5ce0bd1dea9e3a54cd8dcf26849108d8d54e5d5efda6d7bfaacef6d3f4aeb9"}
Jul 27 20:52:31.588387 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.588364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l" event={"ID":"3c8c6773-549b-45e2-8dd5-3eb3441aa67a","Type":"ContainerStarted","Data":"d05af9fa7af4b207aac7eb733c3826908372d7f23a0c6de36825be9fefccc6e2"}
Jul 27 20:52:31.588581 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.588570    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:31.591874 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.591852    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l"
Jul 27 20:52:31.602262 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.602215    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g7ml4" podStartSLOduration=16.799542097 podStartE2EDuration="31.602175293s" totalImagesPullingTime="14.802633196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:52:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:31.60193034 +0000 UTC m=+68.967496187" watchObservedRunningTime="2026-07-27 20:52:31.602175293 +0000 UTC m=+68.967741135"
Jul 27 20:52:31.617551 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.617504    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d8ddb6bf-96z2l" podStartSLOduration=2.495055298 podStartE2EDuration="8.617490294s" totalImagesPullingTime="6.122434996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:31.616969796 +0000 UTC m=+68.982535642" watchObservedRunningTime="2026-07-27 20:52:31.617490294 +0000 UTC m=+68.983056118"
Jul 27 20:52:31.639451 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:31.639352    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54d7b4f96-5pd9h" podStartSLOduration=2.51321073 podStartE2EDuration="8.639336846s" totalImagesPullingTime="6.126126116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:31.638524312 +0000 UTC m=+69.004090158" watchObservedRunningTime="2026-07-27 20:52:31.639336846 +0000 UTC m=+69.004902716"
Jul 27 20:52:32.219600 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:32.219535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.321531 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:32.321499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.435197 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:32.435150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.537307 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:32.537198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.621355 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:32.621322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:32.694617 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:32.694555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:33.225015 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:33.224978    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:33.601096 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:33.601058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmbds" event={"ID":"b322eaf2-56c8-448e-83d0-c4fb746fb0a8","Type":"ContainerStarted","Data":"ae40e7dc2c62ef32f5accfc363b9c0de679916a3c5c80ef79568e98ab7941289"}
Jul 27 20:52:33.601557 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:33.601152    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wmbds" event={"ID":"b322eaf2-56c8-448e-83d0-c4fb746fb0a8","Type":"ContainerStarted","Data":"36f74b23ad7c0ff578d5765b6e73bf42e28e3bc61fb19035cba38d14b8c703de"}
Jul 27 20:52:33.623387 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:33.623326    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wmbds" podStartSLOduration=68.865104004 podStartE2EDuration="1m10.623303872s" totalImagesPullingTime="1.758199868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:33.621282476 +0000 UTC m=+70.986848325" watchObservedRunningTime="2026-07-27 20:52:33.623303872 +0000 UTC m=+70.988869720"
Jul 27 20:52:33.738277 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:33.738213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:33.804734 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:33.804697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:34.813424 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:34.813385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:34.880391 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:34.880354    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:35.411752 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.411718    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d88c6bd4-v84zx"]
Jul 27 20:52:35.414604 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.414572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.424410 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.424385    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 20:52:35.425781 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.425752    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d88c6bd4-v84zx"]
Jul 27 20:52:35.515591 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.515555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-config\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.515765 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.515657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-serving-cert\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.515765 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.515711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-service-ca\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.515765 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.515736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-trusted-ca-bundle\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.515928 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.515776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-oauth-config\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.515928 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.515802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-oauth-serving-cert\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.515928 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.515874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rtt\" (UniqueName: \"kubernetes.io/projected/c4e3afd2-eeda-462c-a85b-fae646ff51cb-kube-api-access-j8rtt\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.597281 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.597242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:35.616389 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.616364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-config\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.616561 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.616413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-serving-cert\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.616634 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.616560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-service-ca\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.616634 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.616600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-trusted-ca-bundle\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.616741 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.616656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-oauth-config\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.616741 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.616684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-oauth-serving-cert\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.616839 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.616732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rtt\" (UniqueName: \"kubernetes.io/projected/c4e3afd2-eeda-462c-a85b-fae646ff51cb-kube-api-access-j8rtt\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.617285 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.617237    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-config\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.617285 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.617253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-service-ca\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.617684 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.617661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-trusted-ca-bundle\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.617845 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.617813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-oauth-serving-cert\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.619694 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.619458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-serving-cert\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.619694 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.619656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-oauth-config\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.627748 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.627726    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rtt\" (UniqueName: \"kubernetes.io/projected/c4e3afd2-eeda-462c-a85b-fae646ff51cb-kube-api-access-j8rtt\") pod \"console-9d88c6bd4-v84zx\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") " pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:35.679592 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.679512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:35.726565 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:35.726530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:37.039546 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.039499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:37.114938 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.114897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:37.620007 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.619936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786676f497-jqk9j" event={"ID":"542fcae9-4569-43c6-8f6b-002e481dda1b","Type":"ContainerStarted","Data":"3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53"}
Jul 27 20:52:37.621519 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.621487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t8lbq" event={"ID":"ab76bb44-d4bb-467c-9c4b-275f4f60b303","Type":"ContainerStarted","Data":"7e1a6ed26e4f6c200766c0b41d36389d8c27d17c52827283024e8e00eea20c38"}
Jul 27 20:52:37.621751 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.621731    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:52:37.644728 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.644652    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-786676f497-jqk9j" podStartSLOduration=1.5255331779999999 podStartE2EDuration="7.64462831s" totalImagesPullingTime="6.119095132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:37.64323786 +0000 UTC m=+75.008803707" watchObservedRunningTime="2026-07-27 20:52:37.64462831 +0000 UTC m=+75.010194157"
Jul 27 20:52:37.649748 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.649715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:37.657884 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.657858    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d88c6bd4-v84zx"]
Jul 27 20:52:37.663861 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.662625    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t8lbq" podStartSLOduration=68.505059881 podStartE2EDuration="1m14.662608538s" totalImagesPullingTime="6.157548657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:37.662177091 +0000 UTC m=+75.027742941" watchObservedRunningTime="2026-07-27 20:52:37.662608538 +0000 UTC m=+75.028174385"
Jul 27 20:52:37.665638 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:37.665600    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e3afd2_eeda_462c_a85b_fae646ff51cb.slice/crio-49cb69291144db866d971828aad872bfedf3c1e494f44fd2ef7cd395666fbb38 WatchSource:0}: Error finding container 49cb69291144db866d971828aad872bfedf3c1e494f44fd2ef7cd395666fbb38: Status 404 returned error can't find the container with id 49cb69291144db866d971828aad872bfedf3c1e494f44fd2ef7cd395666fbb38
Jul 27 20:52:37.702913 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.702874    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"]
Jul 27 20:52:37.705430 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.705400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.710275 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.710251    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 20:52:37.710382 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.710364    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 20:52:37.710623 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.710601    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 20:52:37.710623 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.710618    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 20:52:37.710774 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.710601    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-nnd5g\""
Jul 27 20:52:37.710774 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.710719    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 20:52:37.721064 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.720993    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"]
Jul 27 20:52:37.744396 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.744369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:37.834637 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.834544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.834637 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.834605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.834806 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.834686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8jg\" (UniqueName: \"kubernetes.io/projected/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-kube-api-access-nd8jg\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.834806 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.834719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.935820 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.935785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8jg\" (UniqueName: \"kubernetes.io/projected/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-kube-api-access-nd8jg\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.935820 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.935822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.936052 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.935855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.936052 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.935884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.936052 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:37.935974    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 20:52:37.936052 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:37.936047    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-tls podName:1e1fc0d7-c7bb-4405-8a9c-8c62265b3641 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:38.436025615 +0000 UTC m=+75.801591439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-wdzz2" (UID: "1e1fc0d7-c7bb-4405-8a9c-8c62265b3641") : secret "prometheus-operator-tls" not found
Jul 27 20:52:37.936552 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.936527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.938414 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.938389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:37.944878 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:37.944858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8jg\" (UniqueName: \"kubernetes.io/projected/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-kube-api-access-nd8jg\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:38.016775 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.016739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:38.440825 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.440791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:38.443284 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.443266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1e1fc0d7-c7bb-4405-8a9c-8c62265b3641-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-wdzz2\" (UID: \"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:38.621806 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.621761    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"
Jul 27 20:52:38.626377 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.626342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d88c6bd4-v84zx" event={"ID":"c4e3afd2-eeda-462c-a85b-fae646ff51cb","Type":"ContainerStarted","Data":"ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77"}
Jul 27 20:52:38.626514 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.626401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d88c6bd4-v84zx" event={"ID":"c4e3afd2-eeda-462c-a85b-fae646ff51cb","Type":"ContainerStarted","Data":"49cb69291144db866d971828aad872bfedf3c1e494f44fd2ef7cd395666fbb38"}
Jul 27 20:52:38.627938 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.627903    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4br5p" event={"ID":"59a571a8-5d4f-4cb9-8010-f2257b46dbd3","Type":"ContainerStarted","Data":"9d25ac0213294fcbc1899c339a1558468367a3e4e268b3ac1ceb731f917ed9ec"}
Jul 27 20:52:38.646838 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.646788    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9d88c6bd4-v84zx" podStartSLOduration=3.6467701850000003 podStartE2EDuration="3.646770185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:38.64468507 +0000 UTC m=+76.010250917" watchObservedRunningTime="2026-07-27 20:52:38.646770185 +0000 UTC m=+76.012336032"
Jul 27 20:52:38.660733 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.660677    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4br5p" podStartSLOduration=69.4994311 podStartE2EDuration="1m15.660659842s" totalImagesPullingTime="6.161228742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:38.659566313 +0000 UTC m=+76.025132170" watchObservedRunningTime="2026-07-27 20:52:38.660659842 +0000 UTC m=+76.026225730"
Jul 27 20:52:38.752232 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:38.752204    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-wdzz2"]
Jul 27 20:52:38.755460 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:38.755429    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1fc0d7_c7bb_4405_8a9c_8c62265b3641.slice/crio-1de202e8061883e64f2c22978b438f0f7f0ed8366031847b823ffce47aeedaf9 WatchSource:0}: Error finding container 1de202e8061883e64f2c22978b438f0f7f0ed8366031847b823ffce47aeedaf9: Status 404 returned error can't find the container with id 1de202e8061883e64f2c22978b438f0f7f0ed8366031847b823ffce47aeedaf9
Jul 27 20:52:39.632756 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:39.632724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2" event={"ID":"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641","Type":"ContainerStarted","Data":"1de202e8061883e64f2c22978b438f0f7f0ed8366031847b823ffce47aeedaf9"}
Jul 27 20:52:40.545934 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.545842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:40.546080 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.545949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:40.547578 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.547549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:40.551029 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.551010    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:40.637599 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.637557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2" event={"ID":"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641","Type":"ContainerStarted","Data":"1c356f23c82ee45ec70e8b769c36264894a117540e8b5f0a80f7d1343d20c1d4"}
Jul 27 20:52:40.638020 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.637622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2" event={"ID":"1e1fc0d7-c7bb-4405-8a9c-8c62265b3641","Type":"ContainerStarted","Data":"6a8fc1fb428d5967fad76349ab4caf21dfb75d61276fc4e5ea16f4f6f0600eb2"}
Jul 27 20:52:40.641451 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.641434    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:52:40.656426 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.656376    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-wdzz2" podStartSLOduration=2.145686949 podStartE2EDuration="3.656360381s" totalImagesPullingTime="1.510673432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:40.655012879 +0000 UTC m=+78.020578725" watchObservedRunningTime="2026-07-27 20:52:40.656360381 +0000 UTC m=+78.021926229"
Jul 27 20:52:40.901019 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.900932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:40.995543 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:40.995512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:41.867196 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:41.867161    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:42.067921 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.067886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:42.142829 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.142740    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qqsrq"]
Jul 27 20:52:42.145174 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.145157    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.147899 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.147871    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 20:52:42.148003 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.147937    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 20:52:42.148369 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.148099    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 20:52:42.148369 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.148149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wfrq9\""
Jul 27 20:52:42.197638 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.197604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:42.277561 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.277760 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.277760 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-wtmp\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.277760 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-root\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.277760 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-sys\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.277760 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.278000 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmqc\" (UniqueName: \"kubernetes.io/projected/531d1f22-d6c8-4f48-9add-d4b6729f56a1-kube-api-access-zqmqc\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.278000 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/531d1f22-d6c8-4f48-9add-d4b6729f56a1-metrics-client-ca\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.278000 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.277895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-textfile\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.378753 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.378701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.378753 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.378753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.378778    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-wtmp\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.378807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-root\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.378881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-root\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.378878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-sys\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.378932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-sys\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379014 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:42.379004    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 20:52:42.379014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.379010    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-wtmp\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379371 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.379057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379371 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:42.379077    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls podName:531d1f22-d6c8-4f48-9add-d4b6729f56a1 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:42.879056053 +0000 UTC m=+80.244621878 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls") pod "node-exporter-qqsrq" (UID: "531d1f22-d6c8-4f48-9add-d4b6729f56a1") : secret "node-exporter-tls" not found
Jul 27 20:52:42.379371 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.379113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmqc\" (UniqueName: \"kubernetes.io/projected/531d1f22-d6c8-4f48-9add-d4b6729f56a1-kube-api-access-zqmqc\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379371 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.379162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/531d1f22-d6c8-4f48-9add-d4b6729f56a1-metrics-client-ca\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379371 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.379222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-textfile\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379700 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.379610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-textfile\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.379700 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.379662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.380008 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.379855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/531d1f22-d6c8-4f48-9add-d4b6729f56a1-metrics-client-ca\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.381611 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.381588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.393667 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.393615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmqc\" (UniqueName: \"kubernetes.io/projected/531d1f22-d6c8-4f48-9add-d4b6729f56a1-kube-api-access-zqmqc\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.435260 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.435171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:42.535087 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.535047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:42.635002 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.634969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:42.884426 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.884385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:42.884837 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:42.884506    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 20:52:42.884837 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:52:42.884561    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls podName:531d1f22-d6c8-4f48-9add-d4b6729f56a1 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:43.88454593 +0000 UTC m=+81.250111755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls") pod "node-exporter-qqsrq" (UID: "531d1f22-d6c8-4f48-9add-d4b6729f56a1") : secret "node-exporter-tls" not found
Jul 27 20:52:42.960049 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:42.960013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:43.083317 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.083284    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:43.171488 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.171327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:43.176883 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.176856    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:52:43.180975 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.180955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.184657 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.184637    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 20:52:43.186582 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.186562    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 20:52:43.186662 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.186645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 20:52:43.186809 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.186793    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 20:52:43.186866 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.186849    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 20:52:43.186947 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.186927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mvrlg\""
Jul 27 20:52:43.186947 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.186927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 20:52:43.187106 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.186945    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 20:52:43.187106 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.186927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 20:52:43.187106 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.187091    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 20:52:43.208947 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.208920    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:52:43.288102 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288102 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d01aa4-5e7d-4827-9228-2ae9422e5705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ljr\" (UniqueName: \"kubernetes.io/projected/32d01aa4-5e7d-4827-9228-2ae9422e5705-kube-api-access-t8ljr\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-web-config\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-config-volume\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32d01aa4-5e7d-4827-9228-2ae9422e5705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32d01aa4-5e7d-4827-9228-2ae9422e5705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32d01aa4-5e7d-4827-9228-2ae9422e5705-config-out\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288568 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32d01aa4-5e7d-4827-9228-2ae9422e5705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288568 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288568 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.288568 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.288487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.296040 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.295989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:43.389882 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.389846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.389882 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.389884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.389903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.389921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.389945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.389983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d01aa4-5e7d-4827-9228-2ae9422e5705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.390008    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ljr\" (UniqueName: \"kubernetes.io/projected/32d01aa4-5e7d-4827-9228-2ae9422e5705-kube-api-access-t8ljr\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.390040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-web-config\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.390082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-config-volume\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.390101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32d01aa4-5e7d-4827-9228-2ae9422e5705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390139 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.390137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32d01aa4-5e7d-4827-9228-2ae9422e5705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390587 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.390165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32d01aa4-5e7d-4827-9228-2ae9422e5705-config-out\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390880 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.390712    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32d01aa4-5e7d-4827-9228-2ae9422e5705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.390880 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.390211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32d01aa4-5e7d-4827-9228-2ae9422e5705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.391922 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.391583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32d01aa4-5e7d-4827-9228-2ae9422e5705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.391922 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.391693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32d01aa4-5e7d-4827-9228-2ae9422e5705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.393359 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.393314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.394122 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.394075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-config-volume\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.394122 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.394106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.394299 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.394271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.394343 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.394312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.394642 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.394622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32d01aa4-5e7d-4827-9228-2ae9422e5705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.394727 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.394699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-web-config\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.394802 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.394779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32d01aa4-5e7d-4827-9228-2ae9422e5705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.395634 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.395611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32d01aa4-5e7d-4827-9228-2ae9422e5705-config-out\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.401345 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.401323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ljr\" (UniqueName: \"kubernetes.io/projected/32d01aa4-5e7d-4827-9228-2ae9422e5705-kube-api-access-t8ljr\") pod \"alertmanager-main-0\" (UID: \"32d01aa4-5e7d-4827-9228-2ae9422e5705\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.490964 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.490933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 20:52:43.640090 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.640065    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 20:52:43.641648 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:43.641622    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d01aa4_5e7d_4827_9228_2ae9422e5705.slice/crio-ca34e6df426faa68386bad1d714341229bd39b840d225e3cd7bef93fcda6bd18 WatchSource:0}: Error finding container ca34e6df426faa68386bad1d714341229bd39b840d225e3cd7bef93fcda6bd18: Status 404 returned error can't find the container with id ca34e6df426faa68386bad1d714341229bd39b840d225e3cd7bef93fcda6bd18
Jul 27 20:52:43.649428 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.649402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32d01aa4-5e7d-4827-9228-2ae9422e5705","Type":"ContainerStarted","Data":"ca34e6df426faa68386bad1d714341229bd39b840d225e3cd7bef93fcda6bd18"}
Jul 27 20:52:43.895732 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.895643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:43.898303 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.898270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/531d1f22-d6c8-4f48-9add-d4b6729f56a1-node-exporter-tls\") pod \"node-exporter-qqsrq\" (UID: \"531d1f22-d6c8-4f48-9add-d4b6729f56a1\") " pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:43.954997 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:43.954965    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qqsrq"
Jul 27 20:52:43.965299 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:43.965268    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531d1f22_d6c8_4f48_9add_d4b6729f56a1.slice/crio-c20ba7f5f9e3b272a1acc3a20b6215e937f74bbbeb53aa79e3ae2c837ee230b2 WatchSource:0}: Error finding container c20ba7f5f9e3b272a1acc3a20b6215e937f74bbbeb53aa79e3ae2c837ee230b2: Status 404 returned error can't find the container with id c20ba7f5f9e3b272a1acc3a20b6215e937f74bbbeb53aa79e3ae2c837ee230b2
Jul 27 20:52:44.209641 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.209558    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-696ff8764c-s9mtr"]
Jul 27 20:52:44.214333 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.214214    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.216872 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.216849    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 20:52:44.218008 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.217985    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 20:52:44.218147 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.218023    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b2e73i1k4rifm\""
Jul 27 20:52:44.218147 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.218073    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 20:52:44.218301 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.218212    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6z2l4\""
Jul 27 20:52:44.218301 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.218240    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 20:52:44.218396 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.218346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 20:52:44.229718 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.229697    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-696ff8764c-s9mtr"]
Jul 27 20:52:44.299597 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.299571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.299716 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.299605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.299716 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.299629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-tls\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.299716 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.299676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.299716 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.299710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9978dcf-aeb8-400b-a957-450f83fb6b9a-metrics-client-ca\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.299899 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.299766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.299899 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.299808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-grpc-tls\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.299899 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.299840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fflt\" (UniqueName: \"kubernetes.io/projected/c9978dcf-aeb8-400b-a957-450f83fb6b9a-kube-api-access-7fflt\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.400547 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.400507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9978dcf-aeb8-400b-a957-450f83fb6b9a-metrics-client-ca\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.401043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.400575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.401043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.400620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-grpc-tls\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.401043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.400658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fflt\" (UniqueName: \"kubernetes.io/projected/c9978dcf-aeb8-400b-a957-450f83fb6b9a-kube-api-access-7fflt\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.401043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.400710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.401043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.400749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.401043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.400797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-tls\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.401043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.400834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.402045 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.402006    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9978dcf-aeb8-400b-a957-450f83fb6b9a-metrics-client-ca\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.404440 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.404416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.406239 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.406216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.406887 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.406864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.406975 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.406909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-tls\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.407041 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.407003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-grpc-tls\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.408299 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.408273    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9978dcf-aeb8-400b-a957-450f83fb6b9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.410324 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.410303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fflt\" (UniqueName: \"kubernetes.io/projected/c9978dcf-aeb8-400b-a957-450f83fb6b9a-kube-api-access-7fflt\") pod \"thanos-querier-696ff8764c-s9mtr\" (UID: \"c9978dcf-aeb8-400b-a957-450f83fb6b9a\") " pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.524486 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.524451    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:44.657170 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.657101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqsrq" event={"ID":"531d1f22-d6c8-4f48-9add-d4b6729f56a1","Type":"ContainerStarted","Data":"c20ba7f5f9e3b272a1acc3a20b6215e937f74bbbeb53aa79e3ae2c837ee230b2"}
Jul 27 20:52:44.685739 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.685707    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-696ff8764c-s9mtr"]
Jul 27 20:52:44.689020 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:44.688989    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9978dcf_aeb8_400b_a957_450f83fb6b9a.slice/crio-d688846ce68203c14080a2ef4ed5f4ccabcf7ffa66845cbba92a87723d09c6de WatchSource:0}: Error finding container d688846ce68203c14080a2ef4ed5f4ccabcf7ffa66845cbba92a87723d09c6de: Status 404 returned error can't find the container with id d688846ce68203c14080a2ef4ed5f4ccabcf7ffa66845cbba92a87723d09c6de
Jul 27 20:52:44.716455 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.716422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:44.821229 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.821124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:44.917082 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:44.917046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:45.142158 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.142083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:45.242957 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.242928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:45.612952 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.612910    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:45.662923 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.662885    2577 generic.go:356] "Generic (PLEG): container finished" podID="32d01aa4-5e7d-4827-9228-2ae9422e5705" containerID="1334096deb77ca6e0e3ef49d2502dbd95c4fc7b7593e00276bfa99b79d26dbc8" exitCode=0
Jul 27 20:52:45.663075 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.662971    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32d01aa4-5e7d-4827-9228-2ae9422e5705","Type":"ContainerDied","Data":"1334096deb77ca6e0e3ef49d2502dbd95c4fc7b7593e00276bfa99b79d26dbc8"}
Jul 27 20:52:45.664445 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.664422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr" event={"ID":"c9978dcf-aeb8-400b-a957-450f83fb6b9a","Type":"ContainerStarted","Data":"d688846ce68203c14080a2ef4ed5f4ccabcf7ffa66845cbba92a87723d09c6de"}
Jul 27 20:52:45.666341 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.666314    2577 generic.go:356] "Generic (PLEG): container finished" podID="531d1f22-d6c8-4f48-9add-d4b6729f56a1" containerID="b1211d9a2bab8781a0fbf685794361d087d45cbb05287b131536c43667de3480" exitCode=0
Jul 27 20:52:45.666451 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.666351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqsrq" event={"ID":"531d1f22-d6c8-4f48-9add-d4b6729f56a1","Type":"ContainerDied","Data":"b1211d9a2bab8781a0fbf685794361d087d45cbb05287b131536c43667de3480"}
Jul 27 20:52:45.703577 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.703543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:45.727582 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.727556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:45.727916 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.727894    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:45.733642 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:45.733619    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:46.675929 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:46.675879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqsrq" event={"ID":"531d1f22-d6c8-4f48-9add-d4b6729f56a1","Type":"ContainerStarted","Data":"75393e7d1b6774c0d3fd14fb4e2628390cbfcbd880b8db4b76f175215a8ac283"}
Jul 27 20:52:46.676395 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:46.675968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqsrq" event={"ID":"531d1f22-d6c8-4f48-9add-d4b6729f56a1","Type":"ContainerStarted","Data":"17d364a00b9a091753a5141954853eaf95b2b7fefa8c988ca4e4ed4b7f1ec694"}
Jul 27 20:52:46.680878 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:46.680853    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:52:46.703098 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:46.703044    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qqsrq" podStartSLOduration=3.37855038 podStartE2EDuration="4.703025304s" totalImagesPullingTime="1.252841243s" totalInitContainerRuntime="71.633681ms" isStatefulPod=true podCreationTimestamp="2026-07-27 20:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:46.701619773 +0000 UTC m=+84.067185643" watchObservedRunningTime="2026-07-27 20:52:46.703025304 +0000 UTC m=+84.068591153"
Jul 27 20:52:46.756040 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:46.756012    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786676f497-jqk9j"]
Jul 27 20:52:46.878995 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:46.878966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:46.975297 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:46.975215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:47.473878 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.473813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:47.681088 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.681056    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32d01aa4-5e7d-4827-9228-2ae9422e5705","Type":"ContainerStarted","Data":"8975f8831cd3f4a5d3ecb0885d717fa907e151d0f16fac04b357d734de3aaf82"}
Jul 27 20:52:47.681466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.681142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32d01aa4-5e7d-4827-9228-2ae9422e5705","Type":"ContainerStarted","Data":"09fd44786b9aa4f33e72289bde09fb6794194fe12388a841daef4c711b34a069"}
Jul 27 20:52:47.681466 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.681217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32d01aa4-5e7d-4827-9228-2ae9422e5705","Type":"ContainerStarted","Data":"3c9f8a46419c2cc934a38aeb473e3942ca165f3a9c5c2bf664398050b5f39344"}
Jul 27 20:52:47.683645 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.683623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr" event={"ID":"c9978dcf-aeb8-400b-a957-450f83fb6b9a","Type":"ContainerStarted","Data":"60190561df2c5b18664f71f932bed371f4be8d6c0ba035a75445c1d8bef75abd"}
Jul 27 20:52:47.683748 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.683699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr" event={"ID":"c9978dcf-aeb8-400b-a957-450f83fb6b9a","Type":"ContainerStarted","Data":"a279efd078ce722217c0be0ece31317f41b3c2d85e879fbd31a38496e1dca1e8"}
Jul 27 20:52:47.683748 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.683729    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr" event={"ID":"c9978dcf-aeb8-400b-a957-450f83fb6b9a","Type":"ContainerStarted","Data":"b26e97d2f1651ad0ebc0bbf70d19b2c6bb933d98d0abf908496e6c6d17f0273f"}
Jul 27 20:52:47.692162 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.692140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:47.873777 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:47.873743    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.091904 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.091867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.200654 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.200618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.609002 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.608369    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:52:48.612408 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.612381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.615018 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.614945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:48.617542 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.617513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 20:52:48.617879 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.617856    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 20:52:48.618029 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.617857    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 20:52:48.621992 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.621973    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 20:52:48.623003 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.622976    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 20:52:48.623693 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.623668    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4vltrqeou27qj\""
Jul 27 20:52:48.623793 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.623570    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 20:52:48.623793 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.623533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 20:52:48.623793 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.623596    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 20:52:48.624051 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.624031    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2579h\""
Jul 27 20:52:48.624150 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.624090    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 20:52:48.624232 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.624174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 20:52:48.624356 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.624334    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 20:52:48.624423 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.624344    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 20:52:48.624654 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.624635    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 20:52:48.625620 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.625593    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 20:52:48.628214 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.628176    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 20:52:48.645807 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.645777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:52:48.691528 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.691488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32d01aa4-5e7d-4827-9228-2ae9422e5705","Type":"ContainerStarted","Data":"9957f4d925b3045c3bf53055644ab153f4f18ef2774ae6c26f0b326bfab29965"}
Jul 27 20:52:48.691971 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.691568    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32d01aa4-5e7d-4827-9228-2ae9422e5705","Type":"ContainerStarted","Data":"3cfe6b58789a37b9a3dbebbd64c8d6324533c0ff940fb769a24328019ad6072a"}
Jul 27 20:52:48.691971 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.691609    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32d01aa4-5e7d-4827-9228-2ae9422e5705","Type":"ContainerStarted","Data":"b5dfc20c7f9b10a783ffca88ef3492c86d6a9472a22a2156aab292a166f3d07d"}
Jul 27 20:52:48.698925 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.698897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr" event={"ID":"c9978dcf-aeb8-400b-a957-450f83fb6b9a","Type":"ContainerStarted","Data":"5a526c6fce1f70c4a4e8d0d19b6a694ccdaed40b0026646bd824d0bdbc7b4ac2"}
Jul 27 20:52:48.699169 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.699141    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr" event={"ID":"c9978dcf-aeb8-400b-a957-450f83fb6b9a","Type":"ContainerStarted","Data":"73a385d57fa99ed898db2e7434572f54ec11aaad03b50e2a83342c0f5be89fc3"}
Jul 27 20:52:48.699291 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.699205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr" event={"ID":"c9978dcf-aeb8-400b-a957-450f83fb6b9a","Type":"ContainerStarted","Data":"9d395b10465e211144e01e892c111ac52dac8d5928878beafed6934499b2819e"}
Jul 27 20:52:48.699291 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.699259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:48.736983 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.736928    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr" podStartSLOduration=1.559392667 podStartE2EDuration="4.73690792s" totalImagesPullingTime="3.177515253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:48.73567517 +0000 UTC m=+86.101241044" watchObservedRunningTime="2026-07-27 20:52:48.73690792 +0000 UTC m=+86.102473766"
Jul 27 20:52:48.740249 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4f016d1-a1bf-4ce3-8935-86f591640eb8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740383 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740383 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740383 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kl5q\" (UniqueName: \"kubernetes.io/projected/b4f016d1-a1bf-4ce3-8935-86f591640eb8-kube-api-access-9kl5q\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740383 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740522 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740522 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740522 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4f016d1-a1bf-4ce3-8935-86f591640eb8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740522 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740522 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740676 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740676 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740676 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740782 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740782 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740782 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740782 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-config\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740892 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740892 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.740892 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.740853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.791168 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.791108    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.7888381180000001 podStartE2EDuration="5.791092319s" totalImagesPullingTime="3.985682376s" totalInitContainerRuntime="16.571825ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:48.788722543 +0000 UTC m=+86.154288403" watchObservedRunningTime="2026-07-27 20:52:48.791092319 +0000 UTC m=+86.156658205"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.842698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.842757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.842786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.842816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-config\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.842838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.842908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.842997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4f016d1-a1bf-4ce3-8935-86f591640eb8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843318    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kl5q\" (UniqueName: \"kubernetes.io/projected/b4f016d1-a1bf-4ce3-8935-86f591640eb8-kube-api-access-9kl5q\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.843494 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.844286 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843587    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4f016d1-a1bf-4ce3-8935-86f591640eb8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.844286 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.844286 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.844286 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.844286 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.844286 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.843855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.844790 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.844762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.844944 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.844924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.845224 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.845176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.845648 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.845616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.846314 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.846292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.847014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.846784    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.847014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.846866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.847014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.846960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-config\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.847014 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.846985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.848877 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.848391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.851113 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.851082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4f016d1-a1bf-4ce3-8935-86f591640eb8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.852720 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.852690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.853161 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.853136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.857997 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.857839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.859133 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.858688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4f016d1-a1bf-4ce3-8935-86f591640eb8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.859133 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.858854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4f016d1-a1bf-4ce3-8935-86f591640eb8-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.859360 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.859321    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.860674 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.860650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.860767 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.860737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4f016d1-a1bf-4ce3-8935-86f591640eb8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.861373 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.861352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kl5q\" (UniqueName: \"kubernetes.io/projected/b4f016d1-a1bf-4ce3-8935-86f591640eb8-kube-api-access-9kl5q\") pod \"prometheus-k8s-0\" (UID: \"b4f016d1-a1bf-4ce3-8935-86f591640eb8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:48.924397 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:48.924303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:52:49.076160 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:49.076131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 20:52:49.079478 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:52:49.079444    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f016d1_a1bf_4ce3_8935_86f591640eb8.slice/crio-553a330fce75d8bad1864048762ee7832a0af8074ec1a27e1f2eb69f727943e3 WatchSource:0}: Error finding container 553a330fce75d8bad1864048762ee7832a0af8074ec1a27e1f2eb69f727943e3: Status 404 returned error can't find the container with id 553a330fce75d8bad1864048762ee7832a0af8074ec1a27e1f2eb69f727943e3
Jul 27 20:52:49.703426 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:49.703386    2577 generic.go:356] "Generic (PLEG): container finished" podID="b4f016d1-a1bf-4ce3-8935-86f591640eb8" containerID="bde503675d77df7a0746bac2b64ee088cbd0b308e6a9590dc4901fe778707051" exitCode=0
Jul 27 20:52:49.703896 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:49.703485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4f016d1-a1bf-4ce3-8935-86f591640eb8","Type":"ContainerDied","Data":"bde503675d77df7a0746bac2b64ee088cbd0b308e6a9590dc4901fe778707051"}
Jul 27 20:52:49.703896 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:49.703552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4f016d1-a1bf-4ce3-8935-86f591640eb8","Type":"ContainerStarted","Data":"553a330fce75d8bad1864048762ee7832a0af8074ec1a27e1f2eb69f727943e3"}
Jul 27 20:52:49.801955 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:49.801921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:49.864229 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:49.864204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:51.800696 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:51.800654    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:51.874050 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:51.874012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:51.944581 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:51.944546    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:52.615612 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:52.615576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:53.181415 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:53.181381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:53.285097 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:53.285057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:53.676968 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:53.676922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:53.743709 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:53.743673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:52:54.735021 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:54.734993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4f016d1-a1bf-4ce3-8935-86f591640eb8","Type":"ContainerStarted","Data":"de1e3205fdc65556e5e4c64b28ede4aef11e76989106e5e444cba93d502f9f3f"}
Jul 27 20:52:54.735317 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:54.735086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4f016d1-a1bf-4ce3-8935-86f591640eb8","Type":"ContainerStarted","Data":"ca287d5d1989eadc5118212bad66a65c4db9fe5c008d5b67fd0eb92e5e9cd7c7"}
Jul 27 20:52:54.735317 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:54.735114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4f016d1-a1bf-4ce3-8935-86f591640eb8","Type":"ContainerStarted","Data":"1c6ffbb326ab41b4484b7a1e86b667bfe2937cd3da15a16eed791578634ec9c5"}
Jul 27 20:52:54.735317 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:54.735137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4f016d1-a1bf-4ce3-8935-86f591640eb8","Type":"ContainerStarted","Data":"a680bcb010add8b3605566e9cd378719a69c0d28aad2b1bf5f065c6c147cb7ca"}
Jul 27 20:52:55.714171 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:55.714144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-696ff8764c-s9mtr"
Jul 27 20:52:55.743647 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:55.743615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4f016d1-a1bf-4ce3-8935-86f591640eb8","Type":"ContainerStarted","Data":"f7a9977c4b0654b9875f05e3e06beffce4d7bdc9c14c668e751dc5a610e138d4"}
Jul 27 20:52:55.744026 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:55.743677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4f016d1-a1bf-4ce3-8935-86f591640eb8","Type":"ContainerStarted","Data":"b07ea9dd979df07582e06aa01f9822a54d2dc047a8cd67fcb19f8c899c4462a2"}
Jul 27 20:52:55.778163 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:55.778115    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.287478897 podStartE2EDuration="7.778098216s" totalImagesPullingTime="4.474174381s" totalInitContainerRuntime="16.444938ms" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:55.777070956 +0000 UTC m=+93.142636815" watchObservedRunningTime="2026-07-27 20:52:55.778098216 +0000 UTC m=+93.143664063"
Jul 27 20:52:58.924948 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:52:58.924912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:01.141346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:01.141310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:01.243380 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:01.243341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:02.921493 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:02.921461    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d88c6bd4-v84zx"]
Jul 27 20:53:03.082393 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:03.082354    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:03.190306 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:03.190225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:04.804651 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:04.804606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:04.870657 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:04.870625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:05.649743 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:05.649705    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hml2p_6f7262a1-1f6a-4252-a2dc-99b96a2422c2/serve-healthcheck-canary/0.log"
Jul 27 20:53:08.630621 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:08.630583    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t8lbq"
Jul 27 20:53:11.783082 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:11.783043    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-786676f497-jqk9j" podUID="542fcae9-4569-43c6-8f6b-002e481dda1b" containerName="console" containerID="cri-o://3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53" gracePeriod=15
Jul 27 20:53:12.047737 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.047673    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:53:12.161511 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.161476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-console-config\") pod \"542fcae9-4569-43c6-8f6b-002e481dda1b\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") "
Jul 27 20:53:12.161511 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.161518    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq64v\" (UniqueName: \"kubernetes.io/projected/542fcae9-4569-43c6-8f6b-002e481dda1b-kube-api-access-bq64v\") pod \"542fcae9-4569-43c6-8f6b-002e481dda1b\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") "
Jul 27 20:53:12.161714 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.161594    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-oauth-serving-cert\") pod \"542fcae9-4569-43c6-8f6b-002e481dda1b\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") "
Jul 27 20:53:12.161714 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.161673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-service-ca\") pod \"542fcae9-4569-43c6-8f6b-002e481dda1b\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") "
Jul 27 20:53:12.161714 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.161707    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-oauth-config\") pod \"542fcae9-4569-43c6-8f6b-002e481dda1b\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") "
Jul 27 20:53:12.161860 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.161739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-serving-cert\") pod \"542fcae9-4569-43c6-8f6b-002e481dda1b\" (UID: \"542fcae9-4569-43c6-8f6b-002e481dda1b\") "
Jul 27 20:53:12.162030 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.162001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-console-config" (OuterVolumeSpecName: "console-config") pod "542fcae9-4569-43c6-8f6b-002e481dda1b" (UID: "542fcae9-4569-43c6-8f6b-002e481dda1b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:12.162166 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.162135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-service-ca" (OuterVolumeSpecName: "service-ca") pod "542fcae9-4569-43c6-8f6b-002e481dda1b" (UID: "542fcae9-4569-43c6-8f6b-002e481dda1b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:12.162166 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.162142    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "542fcae9-4569-43c6-8f6b-002e481dda1b" (UID: "542fcae9-4569-43c6-8f6b-002e481dda1b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:12.164092 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.164064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "542fcae9-4569-43c6-8f6b-002e481dda1b" (UID: "542fcae9-4569-43c6-8f6b-002e481dda1b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:53:12.164200 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.164156    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "542fcae9-4569-43c6-8f6b-002e481dda1b" (UID: "542fcae9-4569-43c6-8f6b-002e481dda1b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:53:12.164289 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.164216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542fcae9-4569-43c6-8f6b-002e481dda1b-kube-api-access-bq64v" (OuterVolumeSpecName: "kube-api-access-bq64v") pod "542fcae9-4569-43c6-8f6b-002e481dda1b" (UID: "542fcae9-4569-43c6-8f6b-002e481dda1b"). InnerVolumeSpecName "kube-api-access-bq64v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:53:12.263048 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.263016    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-oauth-serving-cert\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:12.263048 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.263043    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:12.263048 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.263053    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-oauth-config\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:12.263048 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.263061    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/542fcae9-4569-43c6-8f6b-002e481dda1b-console-serving-cert\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:12.263325 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.263070    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/542fcae9-4569-43c6-8f6b-002e481dda1b-console-config\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:12.263325 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.263079    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq64v\" (UniqueName: \"kubernetes.io/projected/542fcae9-4569-43c6-8f6b-002e481dda1b-kube-api-access-bq64v\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:12.807292 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.807250    2577 generic.go:356] "Generic (PLEG): container finished" podID="542fcae9-4569-43c6-8f6b-002e481dda1b" containerID="3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53" exitCode=0
Jul 27 20:53:12.807762 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.807317    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786676f497-jqk9j"
Jul 27 20:53:12.807762 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.807344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786676f497-jqk9j" event={"ID":"542fcae9-4569-43c6-8f6b-002e481dda1b","Type":"ContainerDied","Data":"3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53"}
Jul 27 20:53:12.807762 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.807459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786676f497-jqk9j" event={"ID":"542fcae9-4569-43c6-8f6b-002e481dda1b","Type":"ContainerDied","Data":"a1bebd9bfa08398d029e768a7ea9641e556f71f6e4a067ab0f5d9007b6d088ec"}
Jul 27 20:53:12.807762 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.807491    2577 scope.go:117] "RemoveContainer" containerID="3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53"
Jul 27 20:53:12.816251 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.816227    2577 scope.go:117] "RemoveContainer" containerID="3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53"
Jul 27 20:53:12.816544 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:53:12.816521    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53\": container with ID starting with 3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53 not found: ID does not exist" containerID="3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53"
Jul 27 20:53:12.816625 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.816558    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53"} err="failed to get container status \"3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53\": rpc error: code = NotFound desc = could not find container \"3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53\": container with ID starting with 3f49dfe8f07fcb7e079cb7cc5c59faa023c492f980604503bf0af8a37d5a1e53 not found: ID does not exist"
Jul 27 20:53:12.828459 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.828433    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786676f497-jqk9j"]
Jul 27 20:53:12.832286 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:12.832263    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-786676f497-jqk9j"]
Jul 27 20:53:13.185437 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:13.185360    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="542fcae9-4569-43c6-8f6b-002e481dda1b" path="/var/lib/kubelet/pods/542fcae9-4569-43c6-8f6b-002e481dda1b/volumes"
Jul 27 20:53:19.801512 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:19.801472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:19.865369 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:19.865334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:27.941709 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:27.941664    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9d88c6bd4-v84zx" podUID="c4e3afd2-eeda-462c-a85b-fae646ff51cb" containerName="console" containerID="cri-o://ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77" gracePeriod=15
Jul 27 20:53:28.185564 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.185540    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:53:28.309077 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309028    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-config\") pod \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") "
Jul 27 20:53:28.309077 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-oauth-serving-cert\") pod \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") "
Jul 27 20:53:28.309337 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309128    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8rtt\" (UniqueName: \"kubernetes.io/projected/c4e3afd2-eeda-462c-a85b-fae646ff51cb-kube-api-access-j8rtt\") pod \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") "
Jul 27 20:53:28.309337 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-serving-cert\") pod \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") "
Jul 27 20:53:28.309337 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309171    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-oauth-config\") pod \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") "
Jul 27 20:53:28.309337 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309241    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-service-ca\") pod \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") "
Jul 27 20:53:28.309337 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309300    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-trusted-ca-bundle\") pod \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\" (UID: \"c4e3afd2-eeda-462c-a85b-fae646ff51cb\") "
Jul 27 20:53:28.309568 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309484    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-config" (OuterVolumeSpecName: "console-config") pod "c4e3afd2-eeda-462c-a85b-fae646ff51cb" (UID: "c4e3afd2-eeda-462c-a85b-fae646ff51cb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:28.309568 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309545    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c4e3afd2-eeda-462c-a85b-fae646ff51cb" (UID: "c4e3afd2-eeda-462c-a85b-fae646ff51cb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:28.309666 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309569    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-config\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:28.309810 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c4e3afd2-eeda-462c-a85b-fae646ff51cb" (UID: "c4e3afd2-eeda-462c-a85b-fae646ff51cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:28.309810 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.309773    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-service-ca" (OuterVolumeSpecName: "service-ca") pod "c4e3afd2-eeda-462c-a85b-fae646ff51cb" (UID: "c4e3afd2-eeda-462c-a85b-fae646ff51cb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:28.311645 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.311618    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c4e3afd2-eeda-462c-a85b-fae646ff51cb" (UID: "c4e3afd2-eeda-462c-a85b-fae646ff51cb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:53:28.311725 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.311656    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e3afd2-eeda-462c-a85b-fae646ff51cb-kube-api-access-j8rtt" (OuterVolumeSpecName: "kube-api-access-j8rtt") pod "c4e3afd2-eeda-462c-a85b-fae646ff51cb" (UID: "c4e3afd2-eeda-462c-a85b-fae646ff51cb"). InnerVolumeSpecName "kube-api-access-j8rtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:53:28.311725 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.311692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c4e3afd2-eeda-462c-a85b-fae646ff51cb" (UID: "c4e3afd2-eeda-462c-a85b-fae646ff51cb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:53:28.410645 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.410607    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:28.410645 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.410639    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-trusted-ca-bundle\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:28.410645 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.410649    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4e3afd2-eeda-462c-a85b-fae646ff51cb-oauth-serving-cert\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:28.410645 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.410658    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rtt\" (UniqueName: \"kubernetes.io/projected/c4e3afd2-eeda-462c-a85b-fae646ff51cb-kube-api-access-j8rtt\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:28.410908 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.410667    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-serving-cert\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:28.410908 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.410675    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4e3afd2-eeda-462c-a85b-fae646ff51cb-console-oauth-config\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:53:28.880526 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.880493    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4e3afd2-eeda-462c-a85b-fae646ff51cb" containerID="ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77" exitCode=0
Jul 27 20:53:28.880693 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.880542    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d88c6bd4-v84zx"
Jul 27 20:53:28.880693 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.880570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d88c6bd4-v84zx" event={"ID":"c4e3afd2-eeda-462c-a85b-fae646ff51cb","Type":"ContainerDied","Data":"ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77"}
Jul 27 20:53:28.880693 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.880622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d88c6bd4-v84zx" event={"ID":"c4e3afd2-eeda-462c-a85b-fae646ff51cb","Type":"ContainerDied","Data":"49cb69291144db866d971828aad872bfedf3c1e494f44fd2ef7cd395666fbb38"}
Jul 27 20:53:28.880693 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.880644    2577 scope.go:117] "RemoveContainer" containerID="ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77"
Jul 27 20:53:28.889535 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.889520    2577 scope.go:117] "RemoveContainer" containerID="ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77"
Jul 27 20:53:28.889771 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:53:28.889754    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77\": container with ID starting with ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77 not found: ID does not exist" containerID="ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77"
Jul 27 20:53:28.889810 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.889780    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77"} err="failed to get container status \"ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77\": rpc error: code = NotFound desc = could not find container \"ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77\": container with ID starting with ed94163011bd875716e4281e6942000970eabadbcca15716c7e364017cb20b77 not found: ID does not exist"
Jul 27 20:53:28.902587 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.902561    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d88c6bd4-v84zx"]
Jul 27 20:53:28.907591 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:28.907573    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9d88c6bd4-v84zx"]
Jul 27 20:53:29.090390 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:29.090362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:29.184781 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:29.184706    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e3afd2-eeda-462c-a85b-fae646ff51cb" path="/var/lib/kubelet/pods/c4e3afd2-eeda-462c-a85b-fae646ff51cb/volumes"
Jul 27 20:53:29.187885 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:29.187856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:29.250447 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:29.250418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:48.924545 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:48.924490    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:48.943958 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:48.943931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:48.973731 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:48.973695    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 20:53:51.930850 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:51.930813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:58.393036 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:58.392996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:53:58.466981 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:53:58.466952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:06.511879 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:06.511836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:06.612043 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:06.612005    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:06.711791 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:06.711761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:07.229753 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:07.229705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:07.297066 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:07.297027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:10.337716 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:10.337684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:10.438123 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:10.438095    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:17.408398 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:17.408322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:17.509383 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:17.509349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:17.615160 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:17.615124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:19.809685 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:19.809652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:19.909721 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:19.909689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:35.733381 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:35.733342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:35.834116 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:35.834083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:41.127574 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:41.127536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:54:41.224537 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:54:41.224501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:03.063049 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:03.063012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:03.972338 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:03.972297    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:06.624836 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:06.624802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:06.723768 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:06.723731    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:08.739822 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:08.739785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:08.882438 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:08.882405    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:09.013766 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:09.013678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:11.320956 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:11.320913    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:11.571656 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:11.571569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:13.616238 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:13.616204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:13.682335 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:13.682301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:18.370358 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:18.370328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:18.489083 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:18.489050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:18.976051 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:18.976024    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:31.364686 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:31.364651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:31.419661 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:31.419631    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:36.783527 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:36.783484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:36.883678 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:36.883614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:38.993954 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:38.993925    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:53.434637 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.434605    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:53.621916 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.621867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:53.622862 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.622841    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"]
Jul 27 20:55:53.623172 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.623159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="542fcae9-4569-43c6-8f6b-002e481dda1b" containerName="console"
Jul 27 20:55:53.623234 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.623175    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="542fcae9-4569-43c6-8f6b-002e481dda1b" containerName="console"
Jul 27 20:55:53.623234 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.623214    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e3afd2-eeda-462c-a85b-fae646ff51cb" containerName="console"
Jul 27 20:55:53.623234 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.623220    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e3afd2-eeda-462c-a85b-fae646ff51cb" containerName="console"
Jul 27 20:55:53.623327 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.623268    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="542fcae9-4569-43c6-8f6b-002e481dda1b" containerName="console"
Jul 27 20:55:53.623327 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.623280    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e3afd2-eeda-462c-a85b-fae646ff51cb" containerName="console"
Jul 27 20:55:53.626116 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.626099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.631513 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.631482    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 20:55:53.631626 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.631495    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 20:55:53.632676 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.632642    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lfssj\""
Jul 27 20:55:53.632787 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.632685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 20:55:53.632787 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.632685    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 20:55:53.638968 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.638952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:55:53.653066 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.653039    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"]
Jul 27 20:55:53.747281 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.747248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82624754-9748-48c4-ae54-1646f9c036e7-metrics-cert\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.747454 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.747302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82624754-9748-48c4-ae54-1646f9c036e7-manager-config\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.747454 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.747372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t799q\" (UniqueName: \"kubernetes.io/projected/82624754-9748-48c4-ae54-1646f9c036e7-kube-api-access-t799q\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.747454 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.747410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82624754-9748-48c4-ae54-1646f9c036e7-cert\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.848545 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.848509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t799q\" (UniqueName: \"kubernetes.io/projected/82624754-9748-48c4-ae54-1646f9c036e7-kube-api-access-t799q\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.848545 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.848546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82624754-9748-48c4-ae54-1646f9c036e7-cert\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.848799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.848594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82624754-9748-48c4-ae54-1646f9c036e7-metrics-cert\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.848799 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.848625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82624754-9748-48c4-ae54-1646f9c036e7-manager-config\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.849364 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.849336    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/82624754-9748-48c4-ae54-1646f9c036e7-manager-config\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.851335 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.851304    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82624754-9748-48c4-ae54-1646f9c036e7-metrics-cert\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.851434 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.851407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/82624754-9748-48c4-ae54-1646f9c036e7-cert\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.862500 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.862479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t799q\" (UniqueName: \"kubernetes.io/projected/82624754-9748-48c4-ae54-1646f9c036e7-kube-api-access-t799q\") pod \"lws-controller-manager-58799fcf76-wv6hz\" (UID: \"82624754-9748-48c4-ae54-1646f9c036e7\") " pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:53.934916 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:53.934882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:54.155243 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:55:54.155214    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82624754_9748_48c4_ae54_1646f9c036e7.slice/crio-3cc5b51492ef5c278f8b49e7ce918bf77a4595ee33c8dd5b471df35757353002 WatchSource:0}: Error finding container 3cc5b51492ef5c278f8b49e7ce918bf77a4595ee33c8dd5b471df35757353002: Status 404 returned error can't find the container with id 3cc5b51492ef5c278f8b49e7ce918bf77a4595ee33c8dd5b471df35757353002
Jul 27 20:55:54.171851 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:54.171825    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"]
Jul 27 20:55:54.431772 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:54.431692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz" event={"ID":"82624754-9748-48c4-ae54-1646f9c036e7","Type":"ContainerStarted","Data":"3cc5b51492ef5c278f8b49e7ce918bf77a4595ee33c8dd5b471df35757353002"}
Jul 27 20:55:55.901786 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:55.901741    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:56.143011 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:56.142964    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:56.734509 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:56.734475    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:57.287888 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.287845    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:57.416266 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.416232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:57.445914 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.445877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz" event={"ID":"82624754-9748-48c4-ae54-1646f9c036e7","Type":"ContainerStarted","Data":"8c286adc05167812faafd63f62f911532245ee8f1f6eedc788c3506be28d0a4a"}
Jul 27 20:55:57.446206 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.446177    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:55:57.481245 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.481178    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz" podStartSLOduration=2.0665399 podStartE2EDuration="4.481162379s" totalImagesPullingTime="2.414622479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:57.481021638 +0000 UTC m=+274.846587486" watchObservedRunningTime="2026-07-27 20:55:57.481162379 +0000 UTC m=+274.846728226"
Jul 27 20:55:57.482796 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.482762    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"]
Jul 27 20:55:57.484975 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.484962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.495877 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:55:57.495849    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret"
Jul 27 20:55:57.495974 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:55:57.495890    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" podUID="f864b196-c1f5-446e-8a15-6881ca9876c1" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.495974 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:55:57.495943    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret"
Jul 27 20:55:57.496279 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:55:57.496262    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 27 20:55:57.498596 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:55:57.498574    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-h72pm\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h72pm\"" type="*v1.Secret"
Jul 27 20:55:57.500410 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.500396    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 20:55:57.536056 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.536026    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"]
Jul 27 20:55:57.583013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.582923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4k8\" (UniqueName: \"kubernetes.io/projected/f864b196-c1f5-446e-8a15-6881ca9876c1-kube-api-access-zc4k8\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.583013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.582972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f864b196-c1f5-446e-8a15-6881ca9876c1-manifests\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.583013 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.583003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f864b196-c1f5-446e-8a15-6881ca9876c1-tmp\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.583254 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.583209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f864b196-c1f5-446e-8a15-6881ca9876c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.583254 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.583250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f864b196-c1f5-446e-8a15-6881ca9876c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.684365 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.684326    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4k8\" (UniqueName: \"kubernetes.io/projected/f864b196-c1f5-446e-8a15-6881ca9876c1-kube-api-access-zc4k8\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.684559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.684376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f864b196-c1f5-446e-8a15-6881ca9876c1-manifests\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.684559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.684406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f864b196-c1f5-446e-8a15-6881ca9876c1-tmp\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.684559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.684443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f864b196-c1f5-446e-8a15-6881ca9876c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.684559 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.684467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f864b196-c1f5-446e-8a15-6881ca9876c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.684801 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.684779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f864b196-c1f5-446e-8a15-6881ca9876c1-manifests\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:57.686735 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:57.686717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f864b196-c1f5-446e-8a15-6881ca9876c1-tmp\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:58.110692 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.110660    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:58.268071 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.268035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:58.430054 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.429960    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:58.548400 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.548369    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 20:55:58.557499 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.557465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f864b196-c1f5-446e-8a15-6881ca9876c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:58.557641 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.557522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f864b196-c1f5-446e-8a15-6881ca9876c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:58.572032 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.572009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 20:55:58.577784 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.577752    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4k8\" (UniqueName: \"kubernetes.io/projected/f864b196-c1f5-446e-8a15-6881ca9876c1-kube-api-access-zc4k8\") pod \"opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc\" (UID: \"f864b196-c1f5-446e-8a15-6881ca9876c1\") " pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:58.581638 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.581615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:58.732580 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.732548    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h72pm\""
Jul 27 20:55:58.734902 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.734884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:55:58.813648 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.813624    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 20:55:58.834437 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.834404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:55:58.894617 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:58.894595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"]
Jul 27 20:55:58.897393 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:55:58.897371    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf864b196_c1f5_446e_8a15_6881ca9876c1.slice/crio-287375a56f2d6348316113f6b173cd47c46f708656e5c1759ba6df9e244d4e3e WatchSource:0}: Error finding container 287375a56f2d6348316113f6b173cd47c46f708656e5c1759ba6df9e244d4e3e: Status 404 returned error can't find the container with id 287375a56f2d6348316113f6b173cd47c46f708656e5c1759ba6df9e244d4e3e
Jul 27 20:55:59.458637 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:55:59.458599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc" event={"ID":"f864b196-c1f5-446e-8a15-6881ca9876c1","Type":"ContainerStarted","Data":"287375a56f2d6348316113f6b173cd47c46f708656e5c1759ba6df9e244d4e3e"}
Jul 27 20:56:00.091729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:00.091691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:02.472616 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:02.472580    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc" event={"ID":"f864b196-c1f5-446e-8a15-6881ca9876c1","Type":"ContainerStarted","Data":"470594db5a9c857187dfc44c76d1fc582c59160abd247ea74174088a82458351"}
Jul 27 20:56:03.483240 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:03.483178    2577 generic.go:356] "Generic (PLEG): container finished" podID="f864b196-c1f5-446e-8a15-6881ca9876c1" containerID="470594db5a9c857187dfc44c76d1fc582c59160abd247ea74174088a82458351" exitCode=0
Jul 27 20:56:03.483240 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:03.483215    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc" event={"ID":"f864b196-c1f5-446e-8a15-6881ca9876c1","Type":"ContainerDied","Data":"470594db5a9c857187dfc44c76d1fc582c59160abd247ea74174088a82458351"}
Jul 27 20:56:04.489249 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:04.489217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc" event={"ID":"f864b196-c1f5-446e-8a15-6881ca9876c1","Type":"ContainerStarted","Data":"191a1833dda3ccd3a8996836f2aa370ae5b48529d6416e62e84c3342fc1e8c2b"}
Jul 27 20:56:04.489661 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:04.489330    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:56:08.454171 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:08.454137    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58799fcf76-wv6hz"
Jul 27 20:56:08.501979 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:08.501902    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc" podStartSLOduration=7.315634207 podStartE2EDuration="11.501879645s" totalImagesPullingTime="3.152966988s" totalInitContainerRuntime="1.03327845s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:56:04.526469345 +0000 UTC m=+281.892035191" watchObservedRunningTime="2026-07-27 20:56:08.501879645 +0000 UTC m=+285.867445496"
Jul 27 20:56:09.203537 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:09.203501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:09.417574 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:09.417541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:14.028155 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:14.028119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:14.150474 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:14.150438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:15.495729 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:15.495701    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc"
Jul 27 20:56:17.823876 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:17.823839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:18.703113 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.703083    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"]
Jul 27 20:56:18.710602 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.710579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.715117 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.715093    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 20:56:18.715252 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.715115    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 20:56:18.715421 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.715407    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 20:56:18.729714 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.729690    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sxx96\""
Jul 27 20:56:18.732736 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.732721    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 20:56:18.768382 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.768355    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"]
Jul 27 20:56:18.773930 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.773910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnr9d\" (UniqueName: \"kubernetes.io/projected/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-kube-api-access-xnr9d\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.774019 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.774005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-tls-certs\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.774061 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.774027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-tmp\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.874665 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.874632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-tls-certs\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.874665 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.874662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-tmp\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.875180 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.874697    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnr9d\" (UniqueName: \"kubernetes.io/projected/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-kube-api-access-xnr9d\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.877034 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.877010    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-tmp\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.877264 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.877245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-tls-certs\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:18.913504 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:18.913465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnr9d\" (UniqueName: \"kubernetes.io/projected/aa7add0a-17fa-4a49-83d4-533d5cff0a6c-kube-api-access-xnr9d\") pod \"kube-auth-proxy-6f4ccdc5cb-78mtn\" (UID: \"aa7add0a-17fa-4a49-83d4-533d5cff0a6c\") " pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:19.020057 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:19.020021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"
Jul 27 20:56:19.173840 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:19.173798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn"]
Jul 27 20:56:19.175948 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:56:19.175921    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7add0a_17fa_4a49_83d4_533d5cff0a6c.slice/crio-8ccb8a5f405635d4224a3fbd675dd97f67da048e085ee08fd75b7b0cbb1dc039 WatchSource:0}: Error finding container 8ccb8a5f405635d4224a3fbd675dd97f67da048e085ee08fd75b7b0cbb1dc039: Status 404 returned error can't find the container with id 8ccb8a5f405635d4224a3fbd675dd97f67da048e085ee08fd75b7b0cbb1dc039
Jul 27 20:56:19.375351 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:19.375269    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:19.496965 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:19.496935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:19.548041 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:19.548008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn" event={"ID":"aa7add0a-17fa-4a49-83d4-533d5cff0a6c","Type":"ContainerStarted","Data":"8ccb8a5f405635d4224a3fbd675dd97f67da048e085ee08fd75b7b0cbb1dc039"}
Jul 27 20:56:20.225744 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:20.225712    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:20.598865 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:20.598824    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:20.988144 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:20.988091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:21.091412 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:21.091367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:22.563778 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:22.563734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn" event={"ID":"aa7add0a-17fa-4a49-83d4-533d5cff0a6c","Type":"ContainerStarted","Data":"b2364348d0389286214999bd3e75379c159d3de4a0305b1cbcea65e5762dcf16"}
Jul 27 20:56:22.598083 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:22.598032    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f4ccdc5cb-78mtn" podStartSLOduration=1.890051618 podStartE2EDuration="4.59801773s" totalImagesPullingTime="2.707966112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:56:22.596492927 +0000 UTC m=+299.962058776" watchObservedRunningTime="2026-07-27 20:56:22.59801773 +0000 UTC m=+299.963583578"
Jul 27 20:56:22.635536 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:22.635506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:23.048639 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:23.048614    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 20:56:28.021714 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:28.021679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:28.203805 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:28.203755    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:29.238929 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:29.238898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:29.462825 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:29.462786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:29.637452 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:29.637363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:35.333801 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:35.333765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:35.623423 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:35.623338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:36.028381 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:36.028309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:36.373656 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:36.373573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:36.624131 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:36.624054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:36.813496 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:36.813455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:39.700998 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:39.700963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:56:39.932125 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:56:39.932095    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:19.219107 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:19.219009    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:19.313032 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:19.312987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:19.416488 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:19.416457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:20.218142 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:20.218105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:20.365072 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:20.365037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:48.115701 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:48.115659    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:51.750430 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:51.750384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:53.159115 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:53.159080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:53.527418 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:53.527374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:54.665166 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:54.665122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:54.815885 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:54.815853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:56.132968 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:56.132934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:56.231114 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:56.231078    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:57.684562 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.684526    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"]
Jul 27 20:57:57.687942 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.687925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"
Jul 27 20:57:57.691754 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.691732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 27 20:57:57.691754 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.691745    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vm58t\""
Jul 27 20:57:57.692718 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.692683    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 20:57:57.692878 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.692859    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 20:57:57.701218 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.701196    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"]
Jul 27 20:57:57.755908 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.755871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7vnr\" (UniqueName: \"kubernetes.io/projected/5d005d85-8600-4beb-a199-9385529883b0-kube-api-access-v7vnr\") pod \"dns-operator-controller-manager-864dd78c4c-8d4rc\" (UID: \"5d005d85-8600-4beb-a199-9385529883b0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"
Jul 27 20:57:57.857476 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.857441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7vnr\" (UniqueName: \"kubernetes.io/projected/5d005d85-8600-4beb-a199-9385529883b0-kube-api-access-v7vnr\") pod \"dns-operator-controller-manager-864dd78c4c-8d4rc\" (UID: \"5d005d85-8600-4beb-a199-9385529883b0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"
Jul 27 20:57:57.869350 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.869317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7vnr\" (UniqueName: \"kubernetes.io/projected/5d005d85-8600-4beb-a199-9385529883b0-kube-api-access-v7vnr\") pod \"dns-operator-controller-manager-864dd78c4c-8d4rc\" (UID: \"5d005d85-8600-4beb-a199-9385529883b0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"
Jul 27 20:57:57.997598 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:57.997562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"
Jul 27 20:57:58.132354 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:58.132326    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"]
Jul 27 20:57:58.135164 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:57:58.135140    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d005d85_8600_4beb_a199_9385529883b0.slice/crio-46f6f3612db731c1a724d0918f52558afd5f12cb238d21d9b1fedd8d4e68b56e WatchSource:0}: Error finding container 46f6f3612db731c1a724d0918f52558afd5f12cb238d21d9b1fedd8d4e68b56e: Status 404 returned error can't find the container with id 46f6f3612db731c1a724d0918f52558afd5f12cb238d21d9b1fedd8d4e68b56e
Jul 27 20:57:58.137720 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:58.137704    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:57:58.314723 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:58.314636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:57:58.960067 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:58.960034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc" event={"ID":"5d005d85-8600-4beb-a199-9385529883b0","Type":"ContainerStarted","Data":"46f6f3612db731c1a724d0918f52558afd5f12cb238d21d9b1fedd8d4e68b56e"}
Jul 27 20:57:59.784839 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:59.784808    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"]
Jul 27 20:57:59.788285 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:59.788266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:57:59.791104 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:59.791083    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rzc9j\""
Jul 27 20:57:59.798056 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:59.798036    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"]
Jul 27 20:57:59.877606 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:59.877559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbhj\" (UniqueName: \"kubernetes.io/projected/02f58f8e-3b23-4aea-bad8-7243e9c8412c-kube-api-access-6sbhj\") pod \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" (UID: \"02f58f8e-3b23-4aea-bad8-7243e9c8412c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:57:59.978436 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:59.978404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbhj\" (UniqueName: \"kubernetes.io/projected/02f58f8e-3b23-4aea-bad8-7243e9c8412c-kube-api-access-6sbhj\") pod \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" (UID: \"02f58f8e-3b23-4aea-bad8-7243e9c8412c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:57:59.988652 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:57:59.988627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbhj\" (UniqueName: \"kubernetes.io/projected/02f58f8e-3b23-4aea-bad8-7243e9c8412c-kube-api-access-6sbhj\") pod \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" (UID: \"02f58f8e-3b23-4aea-bad8-7243e9c8412c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:00.102462 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:00.102378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:00.253517 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:00.253486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:00.496895 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:00.496873    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"]
Jul 27 20:58:00.498962 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:58:00.498940    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f58f8e_3b23_4aea_bad8_7243e9c8412c.slice/crio-2cc480b6a4e488c9ccbf1cee9eba8b58413e4ef3ee0122da666bff5de1b02d86 WatchSource:0}: Error finding container 2cc480b6a4e488c9ccbf1cee9eba8b58413e4ef3ee0122da666bff5de1b02d86: Status 404 returned error can't find the container with id 2cc480b6a4e488c9ccbf1cee9eba8b58413e4ef3ee0122da666bff5de1b02d86
Jul 27 20:58:00.972154 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:00.972122    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc" event={"ID":"5d005d85-8600-4beb-a199-9385529883b0","Type":"ContainerStarted","Data":"00fa0c6374dbedde6121921561c2f542c6903f48c62ca4a8d997ab01c8552fa9"}
Jul 27 20:58:00.972349 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:00.972230    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"
Jul 27 20:58:00.973600 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:00.973567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj" event={"ID":"02f58f8e-3b23-4aea-bad8-7243e9c8412c","Type":"ContainerStarted","Data":"2cc480b6a4e488c9ccbf1cee9eba8b58413e4ef3ee0122da666bff5de1b02d86"}
Jul 27 20:58:01.006251 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:01.006134    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc" podStartSLOduration=1.7428018619999999 podStartE2EDuration="4.006116755s" totalImagesPullingTime="2.263314893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:01.003846436 +0000 UTC m=+398.369412283" watchObservedRunningTime="2026-07-27 20:58:01.006116755 +0000 UTC m=+398.371682602"
Jul 27 20:58:02.131165 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:02.131125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:02.225782 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:02.225759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:02.423027 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:02.422937    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:02.688887 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:02.688807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:02.985444 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:02.985409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj" event={"ID":"02f58f8e-3b23-4aea-bad8-7243e9c8412c","Type":"ContainerStarted","Data":"b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78"}
Jul 27 20:58:02.985629 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:02.985512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:03.011202 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:03.011146    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj" podStartSLOduration=2.310049786 podStartE2EDuration="4.011129955s" totalImagesPullingTime="1.701080169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:03.01034261 +0000 UTC m=+400.375908457" watchObservedRunningTime="2026-07-27 20:58:03.011129955 +0000 UTC m=+400.376695802"
Jul 27 20:58:05.494177 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:05.494142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:05.593751 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:05.593717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:05.650365 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:05.650337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:11.981981 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:11.981941    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-8d4rc"
Jul 27 20:58:12.479842 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:12.479807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:12.691696 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:12.691658    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:13.432069 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.432029    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"]
Jul 27 20:58:13.432526 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.432243    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" containerName="manager" containerID="cri-o://b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78" gracePeriod=2
Jul 27 20:58:13.434247 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.434224    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:13.453091 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.453064    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"]
Jul 27 20:58:13.462160 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:58:13.462126    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"02f58f8e-3b23-4aea-bad8-7243e9c8412c\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-7b4jj\": pods \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:13.464973 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:58:13.464950    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:13.552374 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.552350    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"]
Jul 27 20:58:13.552700 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.552689    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" containerName="manager"
Jul 27 20:58:13.552736 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.552702    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" containerName="manager"
Jul 27 20:58:13.552771 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.552755    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" containerName="manager"
Jul 27 20:58:13.555547 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.555528    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"
Jul 27 20:58:13.584721 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.584687    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"]
Jul 27 20:58:13.631935 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:58:13.631899    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:13.676669 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.676645    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:13.682807 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:58:13.682763    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:13.704194 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.704155    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpj2\" (UniqueName: \"kubernetes.io/projected/9aefded3-dd1f-45a3-82d4-f508b9b777d9-kube-api-access-qgpj2\") pod \"limitador-operator-controller-manager-859d7b78bd-c2s7v\" (UID: \"9aefded3-dd1f-45a3-82d4-f508b9b777d9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"
Jul 27 20:58:13.805065 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.805027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sbhj\" (UniqueName: \"kubernetes.io/projected/02f58f8e-3b23-4aea-bad8-7243e9c8412c-kube-api-access-6sbhj\") pod \"02f58f8e-3b23-4aea-bad8-7243e9c8412c\" (UID: \"02f58f8e-3b23-4aea-bad8-7243e9c8412c\") "
Jul 27 20:58:13.805249 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.805167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpj2\" (UniqueName: \"kubernetes.io/projected/9aefded3-dd1f-45a3-82d4-f508b9b777d9-kube-api-access-qgpj2\") pod \"limitador-operator-controller-manager-859d7b78bd-c2s7v\" (UID: \"9aefded3-dd1f-45a3-82d4-f508b9b777d9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"
Jul 27 20:58:13.807273 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.807250    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f58f8e-3b23-4aea-bad8-7243e9c8412c-kube-api-access-6sbhj" (OuterVolumeSpecName: "kube-api-access-6sbhj") pod "02f58f8e-3b23-4aea-bad8-7243e9c8412c" (UID: "02f58f8e-3b23-4aea-bad8-7243e9c8412c"). InnerVolumeSpecName "kube-api-access-6sbhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:58:13.824700 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.824660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpj2\" (UniqueName: \"kubernetes.io/projected/9aefded3-dd1f-45a3-82d4-f508b9b777d9-kube-api-access-qgpj2\") pod \"limitador-operator-controller-manager-859d7b78bd-c2s7v\" (UID: \"9aefded3-dd1f-45a3-82d4-f508b9b777d9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"
Jul 27 20:58:13.865491 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.865463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"
Jul 27 20:58:13.906133 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:13.906100    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbhj\" (UniqueName: \"kubernetes.io/projected/02f58f8e-3b23-4aea-bad8-7243e9c8412c-kube-api-access-6sbhj\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 20:58:14.016541 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:14.016482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"]
Jul 27 20:58:14.018632 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:58:14.018607    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aefded3_dd1f_45a3_82d4_f508b9b777d9.slice/crio-4f593299d97c19ff9af08d3604f1af39b5e48f49ad84a2e4f263f6cda9b98541 WatchSource:0}: Error finding container 4f593299d97c19ff9af08d3604f1af39b5e48f49ad84a2e4f263f6cda9b98541: Status 404 returned error can't find the container with id 4f593299d97c19ff9af08d3604f1af39b5e48f49ad84a2e4f263f6cda9b98541
Jul 27 20:58:14.033631 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:14.033605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v" event={"ID":"9aefded3-dd1f-45a3-82d4-f508b9b777d9","Type":"ContainerStarted","Data":"4f593299d97c19ff9af08d3604f1af39b5e48f49ad84a2e4f263f6cda9b98541"}
Jul 27 20:58:14.034909 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:14.034877    2577 generic.go:356] "Generic (PLEG): container finished" podID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" containerID="b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78" exitCode=0
Jul 27 20:58:14.034997 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:14.034934    2577 scope.go:117] "RemoveContainer" containerID="b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78"
Jul 27 20:58:14.035067 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:14.034933    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:14.037994 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:58:14.037960    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:14.043986 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:14.043942    2577 scope.go:117] "RemoveContainer" containerID="b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78"
Jul 27 20:58:14.044375 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:58:14.044352    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78\": container with ID starting with b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78 not found: ID does not exist" containerID="b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78"
Jul 27 20:58:14.044439 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:14.044385    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78"} err="failed to get container status \"b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78\": rpc error: code = NotFound desc = could not find container \"b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78\": container with ID starting with b8f477e5370c1c26b8cf8cffa6c2cb989be5d7fd7256d45f97e72a2036a03e78 not found: ID does not exist"
Jul 27 20:58:14.057410 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:58:14.057381    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:15.041055 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:15.041022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v" event={"ID":"9aefded3-dd1f-45a3-82d4-f508b9b777d9","Type":"ContainerStarted","Data":"891e4248980260661652225c0adc04f03ec9c7c156c27293580cb3cfb1518186"}
Jul 27 20:58:15.041482 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:15.041105    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"
Jul 27 20:58:15.067131 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:15.067081    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v" podStartSLOduration=2.06706661 podStartE2EDuration="2.06706661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:15.065902498 +0000 UTC m=+412.431468357" watchObservedRunningTime="2026-07-27 20:58:15.06706661 +0000 UTC m=+412.432632457"
Jul 27 20:58:15.070180 ip-10-0-131-91 kubenswrapper[2577]: E0727 20:58:15.070157    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-7b4jj\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-7b4jj"
Jul 27 20:58:15.184944 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:15.184913    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02f58f8e-3b23-4aea-bad8-7243e9c8412c" path="/var/lib/kubelet/pods/02f58f8e-3b23-4aea-bad8-7243e9c8412c/volumes"
Jul 27 20:58:16.090598 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:16.090554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:16.194478 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:16.194448    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:20.787905 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:20.787873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:20.994689 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:20.994655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:25.523047 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:25.523001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:25.623443 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:25.623392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:26.049696 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:26.049670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-c2s7v"
Jul 27 20:58:35.501176 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:35.501138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:35.602346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:35.602309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:35.904111 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:35.903994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:36.005101 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:36.005051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:38.907108 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:38.907006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:39.008064 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:39.008027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:43.115315 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:43.115270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:43.216223 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:43.216133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:44.094831 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:44.094800    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:44.177346 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:44.177304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:46.117542 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.117512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:46.217555 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.217526    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:46.282752 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.282718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:46.292062 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.292029    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 20:58:46.295019 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.295004    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:46.297885 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.297866    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t6w6q\""
Jul 27 20:58:46.297980 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.297882    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 20:58:46.308697 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.308669    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 20:58:46.333126 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.333093    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 20:58:46.390034 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.389932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2e2c3270-06b7-4b2d-b6de-7d240ddc58e9-config-file\") pod \"limitador-limitador-54d6d988d5-5pnw4\" (UID: \"2e2c3270-06b7-4b2d-b6de-7d240ddc58e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:46.390209 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.390053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbr9s\" (UniqueName: \"kubernetes.io/projected/2e2c3270-06b7-4b2d-b6de-7d240ddc58e9-kube-api-access-sbr9s\") pod \"limitador-limitador-54d6d988d5-5pnw4\" (UID: \"2e2c3270-06b7-4b2d-b6de-7d240ddc58e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:46.491312 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.491278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbr9s\" (UniqueName: \"kubernetes.io/projected/2e2c3270-06b7-4b2d-b6de-7d240ddc58e9-kube-api-access-sbr9s\") pod \"limitador-limitador-54d6d988d5-5pnw4\" (UID: \"2e2c3270-06b7-4b2d-b6de-7d240ddc58e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:46.491493 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.491359    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2e2c3270-06b7-4b2d-b6de-7d240ddc58e9-config-file\") pod \"limitador-limitador-54d6d988d5-5pnw4\" (UID: \"2e2c3270-06b7-4b2d-b6de-7d240ddc58e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:46.491987 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.491964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2e2c3270-06b7-4b2d-b6de-7d240ddc58e9-config-file\") pod \"limitador-limitador-54d6d988d5-5pnw4\" (UID: \"2e2c3270-06b7-4b2d-b6de-7d240ddc58e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:46.499787 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.499764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbr9s\" (UniqueName: \"kubernetes.io/projected/2e2c3270-06b7-4b2d-b6de-7d240ddc58e9-kube-api-access-sbr9s\") pod \"limitador-limitador-54d6d988d5-5pnw4\" (UID: \"2e2c3270-06b7-4b2d-b6de-7d240ddc58e9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:46.605222 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.605164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:46.735587 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.735551    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 20:58:46.738457 ip-10-0-131-91 kubenswrapper[2577]: W0727 20:58:46.738433    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2c3270_06b7_4b2d_b6de_7d240ddc58e9.slice/crio-e5d6bed0bd1214ca8c39ea358f468f1b5e8c14c5fc3da7a3765834a1ba1372e4 WatchSource:0}: Error finding container e5d6bed0bd1214ca8c39ea358f468f1b5e8c14c5fc3da7a3765834a1ba1372e4: Status 404 returned error can't find the container with id e5d6bed0bd1214ca8c39ea358f468f1b5e8c14c5fc3da7a3765834a1ba1372e4
Jul 27 20:58:46.949918 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:46.949835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:47.040515 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:47.040479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:47.133040 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:47.133006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:47.178383 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:47.178316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4" event={"ID":"2e2c3270-06b7-4b2d-b6de-7d240ddc58e9","Type":"ContainerStarted","Data":"e5d6bed0bd1214ca8c39ea358f468f1b5e8c14c5fc3da7a3765834a1ba1372e4"}
Jul 27 20:58:47.231640 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:47.231606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:50.193127 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:50.193083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4" event={"ID":"2e2c3270-06b7-4b2d-b6de-7d240ddc58e9","Type":"ContainerStarted","Data":"61430f62da7b7e9ba71480480b7379337c1c06464cccf9c9fb5db067f1b4732d"}
Jul 27 20:58:50.193628 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:50.193226    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:58:50.209015 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:50.208945    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4" podStartSLOduration=1.702531525 podStartE2EDuration="4.208928659s" totalImagesPullingTime="2.506397134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:50.207713435 +0000 UTC m=+447.573279281" watchObservedRunningTime="2026-07-27 20:58:50.208928659 +0000 UTC m=+447.574494532"
Jul 27 20:58:57.322738 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:57.322703    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:57.386764 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:57.386726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:57.527410 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:57.527374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:57.628402 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:57.628326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:59.118292 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:59.118261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:58:59.216040 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:58:59.216002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:01.198742 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:01.198712    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"
Jul 27 20:59:03.541914 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:03.541873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:03.642531 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:03.642494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:07.242133 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:07.242051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:07.341291 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:07.341246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:07.694276 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:07.694174    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:11.196760 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:11.196725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:20.101949 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:20.101904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:20.265815 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:20.265763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:27.063899 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:27.063863    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:27.155589 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:27.155529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:27.209431 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:27.209401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:27.572531 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:27.572496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:30.987074 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:30.987041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:31.083854 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:31.083820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:52.654251 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:52.654218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 20:59:52.725884 ip-10-0-131-91 kubenswrapper[2577]: I0727 20:59:52.725851    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:03.592746 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:03.592710    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:03.799813 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:03.799767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:04.211200 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:04.211146    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:04.340952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:04.340922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:04.899082 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:04.899047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:04.997878 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:04.997847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:05.098337 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:05.098303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:00:15.049131 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:15.049055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:15.151234 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:15.151169    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:23.976105 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:23.976070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:24.595339 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:24.595302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:00:42.450198 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:42.450147    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:43.343411 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:43.343371    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:00:49.985685 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:49.985649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:51.063553 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:51.063514    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:00:53.537697 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:53.537656    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:00:53.640027 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:00:53.639994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:00.192447 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:00.192412    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:05.441118 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:05.441070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:07.155681 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:07.155631    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:01:18.253971 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:18.253935    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:20.216146 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.216111    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"]
Jul 27 21:01:20.219566 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.219550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.223644 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.223620    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 21:01:20.223760 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.223652    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 21:01:20.223760 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.223697    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 21:01:20.223760 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.223708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qcv5g\""
Jul 27 21:01:20.231992 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.231968    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"]
Jul 27 21:01:20.242259 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.242239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:01:20.288694 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.288663    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.288694 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.288698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.288904 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.288729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959b6e53-1748-4811-b3a6-2b5ef87207ef-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.288904 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.288804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.288904 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.288865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.289009 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.288945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h497m\" (UniqueName: \"kubernetes.io/projected/959b6e53-1748-4811-b3a6-2b5ef87207ef-kube-api-access-h497m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390026 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.389972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390026 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.390043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h497m\" (UniqueName: \"kubernetes.io/projected/959b6e53-1748-4811-b3a6-2b5ef87207ef-kube-api-access-h497m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390308 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.390076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390308 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.390093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390308 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.390125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959b6e53-1748-4811-b3a6-2b5ef87207ef-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390308 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.390153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390476 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.390453    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390544 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.390494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.390597 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.390539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.392538 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.392516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959b6e53-1748-4811-b3a6-2b5ef87207ef-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.392823 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.392808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959b6e53-1748-4811-b3a6-2b5ef87207ef-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.398040 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.398013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h497m\" (UniqueName: \"kubernetes.io/projected/959b6e53-1748-4811-b3a6-2b5ef87207ef-kube-api-access-h497m\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-xqv49\" (UID: \"959b6e53-1748-4811-b3a6-2b5ef87207ef\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.527822 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.527779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:20.659082 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.659058    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"]
Jul 27 21:01:20.661282 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:01:20.661251    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod959b6e53_1748_4811_b3a6_2b5ef87207ef.slice/crio-ec32f1b211ee394dae5701011dda0d469a6d288aa8278eb1964d1a62e7bec7df WatchSource:0}: Error finding container ec32f1b211ee394dae5701011dda0d469a6d288aa8278eb1964d1a62e7bec7df: Status 404 returned error can't find the container with id ec32f1b211ee394dae5701011dda0d469a6d288aa8278eb1964d1a62e7bec7df
Jul 27 21:01:20.857456 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.857367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:01:20.858604 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:20.858579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49" event={"ID":"959b6e53-1748-4811-b3a6-2b5ef87207ef","Type":"ContainerStarted","Data":"ec32f1b211ee394dae5701011dda0d469a6d288aa8278eb1964d1a62e7bec7df"}
Jul 27 21:01:21.865007 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:21.864974    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49" event={"ID":"959b6e53-1748-4811-b3a6-2b5ef87207ef","Type":"ContainerStarted","Data":"aef9a8ee125060097f33b45997d71bccae2db93221580e6089740e270c5ade05"}
Jul 27 21:01:21.885444 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:21.885389    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49" podStartSLOduration=0.856756902 podStartE2EDuration="1.885373177s" totalImagesPullingTime="1.028616275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:01:21.883100378 +0000 UTC m=+599.248666226" watchObservedRunningTime="2026-07-27 21:01:21.885373177 +0000 UTC m=+599.250939024"
Jul 27 21:01:30.528061 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:30.528027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:30.530413 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:30.528074    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:30.540408 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:30.540387    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:30.913689 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:30.913613    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-xqv49"
Jul 27 21:01:31.557670 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:31.557636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:01:38.603583 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:01:38.603548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:29.696401 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:02:29.696365    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:29.793093 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:02:29.793059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:30.834600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:02:30.834569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:30.934809 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:02:30.934779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:58.971084 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:02:58.971037    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:02:59.070274 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:02:59.070238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:00.969414 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:03:00.969388    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:03:10.564049 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:03:10.564010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:03:19.960445 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:03:19.960362    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:03:37.052116 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:03:37.052081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:03:37.153357 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:03:37.153323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:01.344140 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:04:01.344106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:38.166115 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:04:38.166081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:04:39.388257 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:04:39.388222    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:01.867629 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:01.867594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:05:14.658681 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:14.658644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:05:18.853450 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:18.853418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:18.952545 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:18.952515    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:19.063955 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:19.063918    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:24.630812 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:24.630776    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:24.736390 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:24.736355    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:24.831478 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:24.831445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:24.962587 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:24.962499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:30.602693 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:30.602655    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:30.702335 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:30.702295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:05:47.255844 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:47.255804    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:05:56.954705 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:05:56.954663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:06:05.750286 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:06:05.750245    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:06:17.462956 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:06:17.462872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:06:29.155168 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:06:29.155131    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:06:34.869239 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:06:34.869198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:06:40.757911 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:06:40.757875    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:06:51.455702 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:06:51.455667    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:07:01.646027 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:07:01.645988    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:07:02.534308 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:07:02.534270    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:07:23.760329 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:07:23.760289    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:07:33.352666 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:07:33.352634    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:07:57.358037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:07:57.357995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:08:08.261503 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:08:08.261465    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:08:19.145664 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:08:19.145625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:08:35.349226 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:08:35.349175    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:08:46.254660 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:08:46.254630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:08:56.020407 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:08:56.020369    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:09:05.661953 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:09:05.661910    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:09:14.753786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:09:14.753703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:09:48.195575 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:09:48.195542    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:11:12.456355 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:11:12.456314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:11:23.258090 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:11:23.258053    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:11:36.050407 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:11:36.050371    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:11:46.571683 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:11:46.571645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:12:03.748787 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:12:03.748755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:12:14.356700 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:12:14.356612    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:12:25.056926 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:12:25.056896    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:12:33.955428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:12:33.955393    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:12:44.859492 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:12:44.859459    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:12:56.953849 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:12:56.953812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:13:04.466510 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:04.466471    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:13:21.559921 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:21.559884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:13:29.124883 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:29.124849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:30.655046 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:30.655010    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:13:36.532309 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.532278    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"]
Jul 27 21:13:36.535736 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.535721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.538446 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.538426    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-379c72fa-kserve-self-signed-certs\""
Jul 27 21:13:36.555263 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.555238    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"]
Jul 27 21:13:36.557963 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.557941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-home\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.558075 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.557975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tls-certs\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.558075 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.558003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-model-cache\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.558206 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.558117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhn5\" (UniqueName: \"kubernetes.io/projected/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-kube-api-access-kbhn5\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.558206 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.558168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-dshm\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.558310 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.558264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tmp-dir\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.615396 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.615364    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:36.659509 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.659471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-home\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.659692 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.659518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tls-certs\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.659692 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.659637    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-model-cache\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.659805 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.659712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhn5\" (UniqueName: \"kubernetes.io/projected/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-kube-api-access-kbhn5\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.659805 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.659757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-dshm\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.659916 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.659802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tmp-dir\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.659916 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.659878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-home\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.660024 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.659970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-model-cache\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.660147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.660126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tmp-dir\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.662094 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.662020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-dshm\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.662230 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.662214    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tls-certs\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.668111 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.668090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhn5\" (UniqueName: \"kubernetes.io/projected/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-kube-api-access-kbhn5\") pod \"e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") " pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.844653 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.844562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:36.976325 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:36.976295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"]
Jul 27 21:13:36.979976 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:13:36.979941    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c6a5959_e004_48a3_a0aa_7e42ecc82db2.slice/crio-ce93abbce7934d5472123103ef322fa3db0f2d95105a7ef298f3461b83e836cd WatchSource:0}: Error finding container ce93abbce7934d5472123103ef322fa3db0f2d95105a7ef298f3461b83e836cd: Status 404 returned error can't find the container with id ce93abbce7934d5472123103ef322fa3db0f2d95105a7ef298f3461b83e836cd
Jul 27 21:13:37.127147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:37.127061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85" event={"ID":"6c6a5959-e004-48a3-a0aa-7e42ecc82db2","Type":"ContainerStarted","Data":"3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284"}
Jul 27 21:13:37.127147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:37.127103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85" event={"ID":"6c6a5959-e004-48a3-a0aa-7e42ecc82db2","Type":"ContainerStarted","Data":"ce93abbce7934d5472123103ef322fa3db0f2d95105a7ef298f3461b83e836cd"}
Jul 27 21:13:37.145264 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:37.145220    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85" podStartSLOduration=1.145204293 podStartE2EDuration="1.145204293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:13:37.144060377 +0000 UTC m=+1334.509626224" watchObservedRunningTime="2026-07-27 21:13:37.145204293 +0000 UTC m=+1334.510770134"
Jul 27 21:13:39.956720 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:39.956689    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:13:46.845370 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:46.845274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:46.845370 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:46.845342    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:46.857768 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:46.857741    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:47.184483 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:47.184408    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:13:49.655107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:49.655072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:13:53.549934 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:13:53.549899    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:14:12.255631 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:12.255594    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"]
Jul 27 21:14:12.276574 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:12.276546    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:13.527742 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:13.527711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:14:16.150626 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:16.150589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:14:27.274967 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.274908    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85" podUID="6c6a5959-e004-48a3-a0aa-7e42ecc82db2" containerName="main" containerID="cri-o://3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284" gracePeriod=45
Jul 27 21:14:27.527670 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.527617    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:14:27.620722 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.620685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-model-cache\") pod \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") "
Jul 27 21:14:27.620722 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.620731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tls-certs\") pod \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") "
Jul 27 21:14:27.620976 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.620762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-home\") pod \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") "
Jul 27 21:14:27.620976 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.620788    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tmp-dir\") pod \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") "
Jul 27 21:14:27.620976 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.620827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbhn5\" (UniqueName: \"kubernetes.io/projected/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-kube-api-access-kbhn5\") pod \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") "
Jul 27 21:14:27.620976 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.620880    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-dshm\") pod \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\" (UID: \"6c6a5959-e004-48a3-a0aa-7e42ecc82db2\") "
Jul 27 21:14:27.621164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.621016    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c6a5959-e004-48a3-a0aa-7e42ecc82db2" (UID: "6c6a5959-e004-48a3-a0aa-7e42ecc82db2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:14:27.621164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.621078    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c6a5959-e004-48a3-a0aa-7e42ecc82db2" (UID: "6c6a5959-e004-48a3-a0aa-7e42ecc82db2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:14:27.621164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.621096    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-home" (OuterVolumeSpecName: "home") pod "6c6a5959-e004-48a3-a0aa-7e42ecc82db2" (UID: "6c6a5959-e004-48a3-a0aa-7e42ecc82db2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:14:27.621164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.621141    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tmp-dir\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:14:27.621164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.621154    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-model-cache\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:14:27.623265 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.623232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-dshm" (OuterVolumeSpecName: "dshm") pod "6c6a5959-e004-48a3-a0aa-7e42ecc82db2" (UID: "6c6a5959-e004-48a3-a0aa-7e42ecc82db2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:14:27.623265 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.623242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-kube-api-access-kbhn5" (OuterVolumeSpecName: "kube-api-access-kbhn5") pod "6c6a5959-e004-48a3-a0aa-7e42ecc82db2" (UID: "6c6a5959-e004-48a3-a0aa-7e42ecc82db2"). InnerVolumeSpecName "kube-api-access-kbhn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:14:27.623419 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.623290    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c6a5959-e004-48a3-a0aa-7e42ecc82db2" (UID: "6c6a5959-e004-48a3-a0aa-7e42ecc82db2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:14:27.722007 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.721970    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhn5\" (UniqueName: \"kubernetes.io/projected/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-kube-api-access-kbhn5\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:14:27.722007 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.722002    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-dshm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:14:27.722007 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.722011    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-tls-certs\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:14:27.722249 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:27.722020    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c6a5959-e004-48a3-a0aa-7e42ecc82db2-home\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:14:28.354916 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.354868    2577 generic.go:356] "Generic (PLEG): container finished" podID="6c6a5959-e004-48a3-a0aa-7e42ecc82db2" containerID="3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284" exitCode=0
Jul 27 21:14:28.355395 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.354945    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"
Jul 27 21:14:28.355395 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.354944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85" event={"ID":"6c6a5959-e004-48a3-a0aa-7e42ecc82db2","Type":"ContainerDied","Data":"3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284"}
Jul 27 21:14:28.355395 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.355086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85" event={"ID":"6c6a5959-e004-48a3-a0aa-7e42ecc82db2","Type":"ContainerDied","Data":"ce93abbce7934d5472123103ef322fa3db0f2d95105a7ef298f3461b83e836cd"}
Jul 27 21:14:28.355395 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.355117    2577 scope.go:117] "RemoveContainer" containerID="3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284"
Jul 27 21:14:28.368293 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.368271    2577 scope.go:117] "RemoveContainer" containerID="3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284"
Jul 27 21:14:28.368614 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:14:28.368583    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284\": container with ID starting with 3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284 not found: ID does not exist" containerID="3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284"
Jul 27 21:14:28.368740 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.368614    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284"} err="failed to get container status \"3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284\": rpc error: code = NotFound desc = could not find container \"3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284\": container with ID starting with 3e70c4740ed0b1230ade50e03cc9ffb432ea22c698285ff394300e27ce221284 not found: ID does not exist"
Jul 27 21:14:28.380922 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.380893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"]
Jul 27 21:14:28.384697 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:28.384673    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-379c72fa-kserve-94bf5989f-pff85"]
Jul 27 21:14:29.184735 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:29.184699    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6a5959-e004-48a3-a0aa-7e42ecc82db2" path="/var/lib/kubelet/pods/6c6a5959-e004-48a3-a0aa-7e42ecc82db2/volumes"
Jul 27 21:14:29.854132 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:29.854097    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:14:37.060333 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:37.060298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:14:53.066735 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:14:53.066699    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:15:00.138595 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.138566    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw"]
Jul 27 21:15:00.139037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.138914    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6a5959-e004-48a3-a0aa-7e42ecc82db2" containerName="main"
Jul 27 21:15:00.139037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.138924    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6a5959-e004-48a3-a0aa-7e42ecc82db2" containerName="main"
Jul 27 21:15:00.139037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.138976    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6a5959-e004-48a3-a0aa-7e42ecc82db2" containerName="main"
Jul 27 21:15:00.141996 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.141980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw"
Jul 27 21:15:00.144655 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.144631    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fvvt\""
Jul 27 21:15:00.148161 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.148137    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw"]
Jul 27 21:15:00.153663 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.153641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw"
Jul 27 21:15:00.283636 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.283611    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw"]
Jul 27 21:15:00.286129 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:15:00.286104    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c1c350_9f1c_43c2_872b_b981b1bbfa92.slice/crio-affaa88e992848d5745af47f19578d02d5e64b8bbbeb7f2cd5b0a0afe6bf09bb WatchSource:0}: Error finding container affaa88e992848d5745af47f19578d02d5e64b8bbbeb7f2cd5b0a0afe6bf09bb: Status 404 returned error can't find the container with id affaa88e992848d5745af47f19578d02d5e64b8bbbeb7f2cd5b0a0afe6bf09bb
Jul 27 21:15:00.288395 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.288378    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:15:00.494245 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:00.494213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw" event={"ID":"11c1c350-9f1c-43c2-872b-b981b1bbfa92","Type":"ContainerStarted","Data":"affaa88e992848d5745af47f19578d02d5e64b8bbbeb7f2cd5b0a0afe6bf09bb"}
Jul 27 21:15:03.054280 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:03.054250    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:15:03.511127 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:03.511091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw" event={"ID":"11c1c350-9f1c-43c2-872b-b981b1bbfa92","Type":"ContainerStarted","Data":"ecf8b953abf60159421b54f266681bd953d27fb0ec9ed231534bc510bdb71d44"}
Jul 27 21:15:03.527003 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:03.526955    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw" podStartSLOduration=0.893996527 podStartE2EDuration="3.526940699s" totalImagesPullingTime="2.632944172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:03.525544413 +0000 UTC m=+1420.891110260" watchObservedRunningTime="2026-07-27 21:15:03.526940699 +0000 UTC m=+1420.892506545"
Jul 27 21:15:04.517199 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:04.517149    2577 generic.go:356] "Generic (PLEG): container finished" podID="11c1c350-9f1c-43c2-872b-b981b1bbfa92" containerID="ecf8b953abf60159421b54f266681bd953d27fb0ec9ed231534bc510bdb71d44" exitCode=0
Jul 27 21:15:04.517580 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:04.517223    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw" event={"ID":"11c1c350-9f1c-43c2-872b-b981b1bbfa92","Type":"ContainerDied","Data":"ecf8b953abf60159421b54f266681bd953d27fb0ec9ed231534bc510bdb71d44"}
Jul 27 21:15:05.650791 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:05.650764    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw"
Jul 27 21:15:06.527623 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:06.527592    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw"
Jul 27 21:15:06.527623 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:06.527600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-wnssw" event={"ID":"11c1c350-9f1c-43c2-872b-b981b1bbfa92","Type":"ContainerDied","Data":"affaa88e992848d5745af47f19578d02d5e64b8bbbeb7f2cd5b0a0afe6bf09bb"}
Jul 27 21:15:06.527831 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:06.527646    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="affaa88e992848d5745af47f19578d02d5e64b8bbbeb7f2cd5b0a0afe6bf09bb"
Jul 27 21:15:12.554454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:12.554413    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:15:20.662445 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:20.662361    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:15:30.361308 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:30.361273    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:15:40.052723 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:40.052684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:15:47.955625 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:47.955586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:15:48.031077 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:48.031038    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:15:53.561749 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:15:53.561715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:16:02.655277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:16:02.655245    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:16:12.758534 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:16:12.758495    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:16:22.455299 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:16:22.455258    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:16:32.452065 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:16:32.452025    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:16:41.656039 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:16:41.656004    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:16:53.652602 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:16:53.652561    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:17:03.852454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:03.852411    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:17:15.144484 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:15.144447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:17:19.385229 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:19.385172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:21.565538 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:21.565501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:17:24.508849 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:24.508801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:36.251481 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:36.251441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:17:46.058877 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:46.058844    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:47.451371 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:47.451338    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:17:49.108299 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:49.108264    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:49.202573 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:49.202537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:49.534814 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:49.534777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:54.237379 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:54.237340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:55.337559 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:55.337527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:55.399375 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:55.399338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:55.569086 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:55.569051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:55.900248 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:55.900214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:56.031863 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:56.031800    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:57.925120 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:57.925085    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:58.026697 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.026663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:58.138106 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.138067    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:58.737847 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.737814    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"]
Jul 27 21:17:58.738177 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.738165    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c1c350-9f1c-43c2-872b-b981b1bbfa92" containerName="cleanup"
Jul 27 21:17:58.738177 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.738179    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c1c350-9f1c-43c2-872b-b981b1bbfa92" containerName="cleanup"
Jul 27 21:17:58.738286 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.738260    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c1c350-9f1c-43c2-872b-b981b1bbfa92" containerName="cleanup"
Jul 27 21:17:58.741160 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.741145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:17:58.747478 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.744229    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5f8eb8aa\""
Jul 27 21:17:58.747478 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.744333    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5f8eb8aa-dockercfg-s5pbx\""
Jul 27 21:17:58.755889 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.755864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"]
Jul 27 21:17:58.790967 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.790937    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"]
Jul 27 21:17:58.794332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.794319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:17:58.805542 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.805520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"]
Jul 27 21:17:58.843055 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.843025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d333e6f6-1f15-472d-a766-6c6f4643dd53-plugins-config-volume\") pod \"payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz\" (UID: \"d333e6f6-1f15-472d-a766-6c6f4643dd53\") " pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:17:58.843055 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.843060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdcv2\" (UniqueName: \"kubernetes.io/projected/d333e6f6-1f15-472d-a766-6c6f4643dd53-kube-api-access-kdcv2\") pod \"payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz\" (UID: \"d333e6f6-1f15-472d-a766-6c6f4643dd53\") " pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:17:58.843280 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.843107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwfpz\" (UniqueName: \"kubernetes.io/projected/0b847803-6135-4ea0-8a51-083f639817e6-kube-api-access-xwfpz\") pod \"payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw\" (UID: \"0b847803-6135-4ea0-8a51-083f639817e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:17:58.843280 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.843161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b847803-6135-4ea0-8a51-083f639817e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw\" (UID: \"0b847803-6135-4ea0-8a51-083f639817e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:17:58.849135 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.849108    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:58.944305 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.944279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwfpz\" (UniqueName: \"kubernetes.io/projected/0b847803-6135-4ea0-8a51-083f639817e6-kube-api-access-xwfpz\") pod \"payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw\" (UID: \"0b847803-6135-4ea0-8a51-083f639817e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:17:58.944648 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.944339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b847803-6135-4ea0-8a51-083f639817e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw\" (UID: \"0b847803-6135-4ea0-8a51-083f639817e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:17:58.944648 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.944372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d333e6f6-1f15-472d-a766-6c6f4643dd53-plugins-config-volume\") pod \"payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz\" (UID: \"d333e6f6-1f15-472d-a766-6c6f4643dd53\") " pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:17:58.944648 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.944390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdcv2\" (UniqueName: \"kubernetes.io/projected/d333e6f6-1f15-472d-a766-6c6f4643dd53-kube-api-access-kdcv2\") pod \"payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz\" (UID: \"d333e6f6-1f15-472d-a766-6c6f4643dd53\") " pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:17:58.945280 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.945258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d333e6f6-1f15-472d-a766-6c6f4643dd53-plugins-config-volume\") pod \"payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz\" (UID: \"d333e6f6-1f15-472d-a766-6c6f4643dd53\") " pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:17:58.945321 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.945258    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b847803-6135-4ea0-8a51-083f639817e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw\" (UID: \"0b847803-6135-4ea0-8a51-083f639817e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:17:58.948440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.948414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:17:58.952748 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.952727    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwfpz\" (UniqueName: \"kubernetes.io/projected/0b847803-6135-4ea0-8a51-083f639817e6-kube-api-access-xwfpz\") pod \"payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw\" (UID: \"0b847803-6135-4ea0-8a51-083f639817e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:17:58.952837 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:58.952813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdcv2\" (UniqueName: \"kubernetes.io/projected/d333e6f6-1f15-472d-a766-6c6f4643dd53-kube-api-access-kdcv2\") pod \"payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz\" (UID: \"d333e6f6-1f15-472d-a766-6c6f4643dd53\") " pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:17:59.055742 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:59.055640    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:17:59.104020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:59.103990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:17:59.199023 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:59.198995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"]
Jul 27 21:17:59.203107 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:17:59.203077    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd333e6f6_1f15_472d_a766_6c6f4643dd53.slice/crio-a9bfb435536b234d44e01db5fe10d3bb5890d0d725b1a3847d361ad8785dfd95 WatchSource:0}: Error finding container a9bfb435536b234d44e01db5fe10d3bb5890d0d725b1a3847d361ad8785dfd95: Status 404 returned error can't find the container with id a9bfb435536b234d44e01db5fe10d3bb5890d0d725b1a3847d361ad8785dfd95
Jul 27 21:17:59.251760 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:59.251719    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"]
Jul 27 21:17:59.253967 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:17:59.253943    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b847803_6135_4ea0_8a51_083f639817e6.slice/crio-054b7a6046b014431696fcc6539d53503b619b392af9815837adb3d1d26c5a36 WatchSource:0}: Error finding container 054b7a6046b014431696fcc6539d53503b619b392af9815837adb3d1d26c5a36: Status 404 returned error can't find the container with id 054b7a6046b014431696fcc6539d53503b619b392af9815837adb3d1d26c5a36
Jul 27 21:17:59.296115 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:59.296078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz" event={"ID":"d333e6f6-1f15-472d-a766-6c6f4643dd53","Type":"ContainerStarted","Data":"a9bfb435536b234d44e01db5fe10d3bb5890d0d725b1a3847d361ad8785dfd95"}
Jul 27 21:17:59.297141 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:17:59.297119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw" event={"ID":"0b847803-6135-4ea0-8a51-083f639817e6","Type":"ContainerStarted","Data":"054b7a6046b014431696fcc6539d53503b619b392af9815837adb3d1d26c5a36"}
Jul 27 21:18:01.063076 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:01.063040    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:18:02.315847 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:02.315806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw" event={"ID":"0b847803-6135-4ea0-8a51-083f639817e6","Type":"ContainerStarted","Data":"a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5"}
Jul 27 21:18:02.316362 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:02.315899    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:18:02.317332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:02.317308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz" event={"ID":"d333e6f6-1f15-472d-a766-6c6f4643dd53","Type":"ContainerStarted","Data":"7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4"}
Jul 27 21:18:02.317447 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:02.317365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:18:02.332300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:02.332247    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw" podStartSLOduration=2.00132804 podStartE2EDuration="4.332231549s" totalImagesPullingTime="2.330903509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:02.330551588 +0000 UTC m=+1599.696117435" watchObservedRunningTime="2026-07-27 21:18:02.332231549 +0000 UTC m=+1599.697797396"
Jul 27 21:18:02.347464 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:02.347414    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz" podStartSLOduration=1.9719326179999999 podStartE2EDuration="4.347395914s" totalImagesPullingTime="2.375463296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:02.34556042 +0000 UTC m=+1599.711126267" watchObservedRunningTime="2026-07-27 21:18:02.347395914 +0000 UTC m=+1599.712961760"
Jul 27 21:18:13.324319 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:13.324243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:18:13.324770 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:13.324319    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:18:23.496233 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.496179    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"]
Jul 27 21:18:23.499407 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.499391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"
Jul 27 21:18:23.502094 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.502069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:18:23.502231 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.502125    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fvvt\""
Jul 27 21:18:23.508390 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.508367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"]
Jul 27 21:18:23.571603 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.571569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6c65c3-4d49-4197-ad5d-a76495b8629a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv\" (UID: \"0b6c65c3-4d49-4197-ad5d-a76495b8629a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"
Jul 27 21:18:23.673091 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.673056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6c65c3-4d49-4197-ad5d-a76495b8629a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv\" (UID: \"0b6c65c3-4d49-4197-ad5d-a76495b8629a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"
Jul 27 21:18:23.673689 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.673669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6c65c3-4d49-4197-ad5d-a76495b8629a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv\" (UID: \"0b6c65c3-4d49-4197-ad5d-a76495b8629a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"
Jul 27 21:18:23.811404 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.811320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"
Jul 27 21:18:23.940847 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:23.940821    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"]
Jul 27 21:18:23.942722 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:18:23.942699    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6c65c3_4d49_4197_ad5d_a76495b8629a.slice/crio-d2d16fee6ed619ff1f51eba068dc30e37e54d8ac068aca37fe0270a2a3a12213 WatchSource:0}: Error finding container d2d16fee6ed619ff1f51eba068dc30e37e54d8ac068aca37fe0270a2a3a12213: Status 404 returned error can't find the container with id d2d16fee6ed619ff1f51eba068dc30e37e54d8ac068aca37fe0270a2a3a12213
Jul 27 21:18:24.421272 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:24.421232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv" event={"ID":"0b6c65c3-4d49-4197-ad5d-a76495b8629a","Type":"ContainerStarted","Data":"e6bab33f6d8b11bd52a1d6a28329c7b89169b550aeb2b7d16890d861f20fc2fc"}
Jul 27 21:18:24.421443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:24.421295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv" event={"ID":"0b6c65c3-4d49-4197-ad5d-a76495b8629a","Type":"ContainerStarted","Data":"d2d16fee6ed619ff1f51eba068dc30e37e54d8ac068aca37fe0270a2a3a12213"}
Jul 27 21:18:24.439015 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:24.438908    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv" podStartSLOduration=1.4388931139999999 podStartE2EDuration="1.438893114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:24.436998224 +0000 UTC m=+1621.802564071" watchObservedRunningTime="2026-07-27 21:18:24.438893114 +0000 UTC m=+1621.804458960"
Jul 27 21:18:25.427402 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:25.427369    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b6c65c3-4d49-4197-ad5d-a76495b8629a" containerID="e6bab33f6d8b11bd52a1d6a28329c7b89169b550aeb2b7d16890d861f20fc2fc" exitCode=0
Jul 27 21:18:25.427821 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:25.427428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv" event={"ID":"0b6c65c3-4d49-4197-ad5d-a76495b8629a","Type":"ContainerDied","Data":"e6bab33f6d8b11bd52a1d6a28329c7b89169b550aeb2b7d16890d861f20fc2fc"}
Jul 27 21:18:26.568901 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:26.568878    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"
Jul 27 21:18:26.597962 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:26.597930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6c65c3-4d49-4197-ad5d-a76495b8629a-maas-api-service-ca\") pod \"0b6c65c3-4d49-4197-ad5d-a76495b8629a\" (UID: \"0b6c65c3-4d49-4197-ad5d-a76495b8629a\") "
Jul 27 21:18:26.598337 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:26.598309    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6c65c3-4d49-4197-ad5d-a76495b8629a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b6c65c3-4d49-4197-ad5d-a76495b8629a" (UID: "0b6c65c3-4d49-4197-ad5d-a76495b8629a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:26.698525 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:26.698448    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b6c65c3-4d49-4197-ad5d-a76495b8629a-maas-api-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:18:27.438412 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:27.438382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"
Jul 27 21:18:27.438573 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:27.438419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv" event={"ID":"0b6c65c3-4d49-4197-ad5d-a76495b8629a","Type":"ContainerDied","Data":"d2d16fee6ed619ff1f51eba068dc30e37e54d8ac068aca37fe0270a2a3a12213"}
Jul 27 21:18:27.438573 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:27.438473    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2d16fee6ed619ff1f51eba068dc30e37e54d8ac068aca37fe0270a2a3a12213"
Jul 27 21:18:34.013892 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.013859    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:34.039521 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.039488    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"]
Jul 27 21:18:34.039788 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.039757    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz" podUID="d333e6f6-1f15-472d-a766-6c6f4643dd53" containerName="payload-processing" containerID="cri-o://7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4" gracePeriod=30
Jul 27 21:18:34.050091 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.050053    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"]
Jul 27 21:18:34.050325 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.050271    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw" podUID="0b847803-6135-4ea0-8a51-083f639817e6" containerName="payload-pre-processing" containerID="cri-o://a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5" gracePeriod=30
Jul 27 21:18:34.323395 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.323366    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:18:34.328460 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.328438    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:18:34.364734 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.364705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b847803-6135-4ea0-8a51-083f639817e6-plugins-config-volume\") pod \"0b847803-6135-4ea0-8a51-083f639817e6\" (UID: \"0b847803-6135-4ea0-8a51-083f639817e6\") "
Jul 27 21:18:34.364901 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.364765    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdcv2\" (UniqueName: \"kubernetes.io/projected/d333e6f6-1f15-472d-a766-6c6f4643dd53-kube-api-access-kdcv2\") pod \"d333e6f6-1f15-472d-a766-6c6f4643dd53\" (UID: \"d333e6f6-1f15-472d-a766-6c6f4643dd53\") "
Jul 27 21:18:34.364901 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.364794    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d333e6f6-1f15-472d-a766-6c6f4643dd53-plugins-config-volume\") pod \"d333e6f6-1f15-472d-a766-6c6f4643dd53\" (UID: \"d333e6f6-1f15-472d-a766-6c6f4643dd53\") "
Jul 27 21:18:34.364901 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.364880    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwfpz\" (UniqueName: \"kubernetes.io/projected/0b847803-6135-4ea0-8a51-083f639817e6-kube-api-access-xwfpz\") pod \"0b847803-6135-4ea0-8a51-083f639817e6\" (UID: \"0b847803-6135-4ea0-8a51-083f639817e6\") "
Jul 27 21:18:34.365227 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.365176    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b847803-6135-4ea0-8a51-083f639817e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b847803-6135-4ea0-8a51-083f639817e6" (UID: "0b847803-6135-4ea0-8a51-083f639817e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:34.365331 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.365259    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d333e6f6-1f15-472d-a766-6c6f4643dd53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d333e6f6-1f15-472d-a766-6c6f4643dd53" (UID: "d333e6f6-1f15-472d-a766-6c6f4643dd53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:18:34.367214 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.367170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b847803-6135-4ea0-8a51-083f639817e6-kube-api-access-xwfpz" (OuterVolumeSpecName: "kube-api-access-xwfpz") pod "0b847803-6135-4ea0-8a51-083f639817e6" (UID: "0b847803-6135-4ea0-8a51-083f639817e6"). InnerVolumeSpecName "kube-api-access-xwfpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:18:34.367299 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.367235    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d333e6f6-1f15-472d-a766-6c6f4643dd53-kube-api-access-kdcv2" (OuterVolumeSpecName: "kube-api-access-kdcv2") pod "d333e6f6-1f15-472d-a766-6c6f4643dd53" (UID: "d333e6f6-1f15-472d-a766-6c6f4643dd53"). InnerVolumeSpecName "kube-api-access-kdcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:18:34.466085 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.466048    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwfpz\" (UniqueName: \"kubernetes.io/projected/0b847803-6135-4ea0-8a51-083f639817e6-kube-api-access-xwfpz\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:18:34.466085 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.466080    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b847803-6135-4ea0-8a51-083f639817e6-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:18:34.466341 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.466093    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdcv2\" (UniqueName: \"kubernetes.io/projected/d333e6f6-1f15-472d-a766-6c6f4643dd53-kube-api-access-kdcv2\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:18:34.466341 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.466108    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d333e6f6-1f15-472d-a766-6c6f4643dd53-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:18:34.470741 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.470710    2577 generic.go:356] "Generic (PLEG): container finished" podID="d333e6f6-1f15-472d-a766-6c6f4643dd53" containerID="7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4" exitCode=0
Jul 27 21:18:34.470864 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.470758    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"
Jul 27 21:18:34.470864 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.470802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz" event={"ID":"d333e6f6-1f15-472d-a766-6c6f4643dd53","Type":"ContainerDied","Data":"7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4"}
Jul 27 21:18:34.470938 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.470888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz" event={"ID":"d333e6f6-1f15-472d-a766-6c6f4643dd53","Type":"ContainerDied","Data":"a9bfb435536b234d44e01db5fe10d3bb5890d0d725b1a3847d361ad8785dfd95"}
Jul 27 21:18:34.470938 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.470922    2577 scope.go:117] "RemoveContainer" containerID="7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4"
Jul 27 21:18:34.472257 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.472224    2577 generic.go:356] "Generic (PLEG): container finished" podID="0b847803-6135-4ea0-8a51-083f639817e6" containerID="a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5" exitCode=0
Jul 27 21:18:34.472341 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.472265    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"
Jul 27 21:18:34.472341 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.472289    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw" event={"ID":"0b847803-6135-4ea0-8a51-083f639817e6","Type":"ContainerDied","Data":"a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5"}
Jul 27 21:18:34.472341 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.472334    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw" event={"ID":"0b847803-6135-4ea0-8a51-083f639817e6","Type":"ContainerDied","Data":"054b7a6046b014431696fcc6539d53503b619b392af9815837adb3d1d26c5a36"}
Jul 27 21:18:34.482573 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.482555    2577 scope.go:117] "RemoveContainer" containerID="7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4"
Jul 27 21:18:34.482865 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:18:34.482840    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4\": container with ID starting with 7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4 not found: ID does not exist" containerID="7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4"
Jul 27 21:18:34.482929 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.482875    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4"} err="failed to get container status \"7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4\": rpc error: code = NotFound desc = could not find container \"7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4\": container with ID starting with 7b4ebe7cc0981299fbfbe8333a0a74c85d0298d61e91382a05139c4f0a22cbb4 not found: ID does not exist"
Jul 27 21:18:34.482929 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.482892    2577 scope.go:117] "RemoveContainer" containerID="a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5"
Jul 27 21:18:34.490998 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.490983    2577 scope.go:117] "RemoveContainer" containerID="a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5"
Jul 27 21:18:34.491299 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:18:34.491280    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5\": container with ID starting with a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5 not found: ID does not exist" containerID="a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5"
Jul 27 21:18:34.491367 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.491306    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5"} err="failed to get container status \"a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5\": rpc error: code = NotFound desc = could not find container \"a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5\": container with ID starting with a77710cc431b2bb6ecc1928e3f1489550f4e141604405ddb091b21192d65fcc5 not found: ID does not exist"
Jul 27 21:18:34.499690 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.499668    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"]
Jul 27 21:18:34.503247 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.503224    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5f8eb8aa-5679cb959b-qshjz"]
Jul 27 21:18:34.514001 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.513981    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"]
Jul 27 21:18:34.517885 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:34.517865    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5f8eb8aa-fff795456-9hhdw"]
Jul 27 21:18:35.184555 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:35.184523    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b847803-6135-4ea0-8a51-083f639817e6" path="/var/lib/kubelet/pods/0b847803-6135-4ea0-8a51-083f639817e6/volumes"
Jul 27 21:18:35.184935 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:35.184860    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d333e6f6-1f15-472d-a766-6c6f4643dd53" path="/var/lib/kubelet/pods/d333e6f6-1f15-472d-a766-6c6f4643dd53/volumes"
Jul 27 21:18:35.544505 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:35.544468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:37.050081 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:37.050047    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:18:38.580291 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:38.580255    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"]
Jul 27 21:18:38.584121 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:38.584097    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5f8eb8aa-vwjxv"]
Jul 27 21:18:38.587722 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:38.587682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:38.731607 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:38.731563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:38.950515 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:38.950429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:39.184590 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:39.184557    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6c65c3-4d49-4197-ad5d-a76495b8629a" path="/var/lib/kubelet/pods/0b6c65c3-4d49-4197-ad5d-a76495b8629a/volumes"
Jul 27 21:18:39.185717 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:39.185690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:39.282602 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:39.282573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:39.386384 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:39.386349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:39.791519 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:39.791476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:39.955565 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:39.955534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:42.269727 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.269685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:42.346345 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346313    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"]
Jul 27 21:18:42.346709 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346697    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b847803-6135-4ea0-8a51-083f639817e6" containerName="payload-pre-processing"
Jul 27 21:18:42.346755 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346712    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b847803-6135-4ea0-8a51-083f639817e6" containerName="payload-pre-processing"
Jul 27 21:18:42.346755 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346727    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d333e6f6-1f15-472d-a766-6c6f4643dd53" containerName="payload-processing"
Jul 27 21:18:42.346755 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346734    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d333e6f6-1f15-472d-a766-6c6f4643dd53" containerName="payload-processing"
Jul 27 21:18:42.346755 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346748    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6c65c3-4d49-4197-ad5d-a76495b8629a" containerName="revoke-keys"
Jul 27 21:18:42.346755 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346754    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6c65c3-4d49-4197-ad5d-a76495b8629a" containerName="revoke-keys"
Jul 27 21:18:42.346942 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346812    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b847803-6135-4ea0-8a51-083f639817e6" containerName="payload-pre-processing"
Jul 27 21:18:42.346942 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346825    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6c65c3-4d49-4197-ad5d-a76495b8629a" containerName="revoke-keys"
Jul 27 21:18:42.346942 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.346831    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d333e6f6-1f15-472d-a766-6c6f4643dd53" containerName="payload-processing"
Jul 27 21:18:42.351040 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.351020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.353638 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.353616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-20b824d1-serving-cert\""
Jul 27 21:18:42.353777 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.353655    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:18:42.353777 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.353654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cgjq6\""
Jul 27 21:18:42.354624 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.354606    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:18:42.361800 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.361777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"]
Jul 27 21:18:42.389935 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.389901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:42.429290 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.429256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdqh\" (UniqueName: \"kubernetes.io/projected/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-kube-api-access-2mdqh\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.429290 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.429295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.429485 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.429334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-openshift-service-ca\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.529905 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.529825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-openshift-service-ca\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.529905 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.529890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdqh\" (UniqueName: \"kubernetes.io/projected/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-kube-api-access-2mdqh\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.530099 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.529913    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.530099 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:18:42.530002    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-serving-cert: secret "maas-api-e2e-migrate-20b824d1-serving-cert" not found
Jul 27 21:18:42.530099 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:18:42.530063    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls podName:4b4a3f75-b3f8-4f38-a4cf-a624001d7214 nodeName:}" failed. No retries permitted until 2026-07-27 21:18:43.030047797 +0000 UTC m=+1640.395613623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls") pod "maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk" (UID: "4b4a3f75-b3f8-4f38-a4cf-a624001d7214") : secret "maas-api-e2e-migrate-20b824d1-serving-cert" not found
Jul 27 21:18:42.530583 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.530564    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-openshift-service-ca\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.538679 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.538658    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdqh\" (UniqueName: \"kubernetes.io/projected/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-kube-api-access-2mdqh\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:42.948567 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:42.948479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:43.034489 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.034456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:43.036993 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.036969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls\") pod \"maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:43.062674 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.062623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:43.166244 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.166215    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"]
Jul 27 21:18:43.166890 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.166856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:43.168630 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.168613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:43.171124 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.171100    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-20b824d1\""
Jul 27 21:18:43.171260 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.171166    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-20b824d1-dockercfg-lz8mp\""
Jul 27 21:18:43.185918 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.185894    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"]
Jul 27 21:18:43.196761 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.196732    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"]
Jul 27 21:18:43.199107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.199046    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:43.211945 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.211918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"]
Jul 27 21:18:43.236499 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.236464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l7hp\" (UniqueName: \"kubernetes.io/projected/393eee2a-9aa2-465a-b56c-e97fe676bb63-kube-api-access-2l7hp\") pod \"payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v\" (UID: \"393eee2a-9aa2-465a-b56c-e97fe676bb63\") " pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:43.236639 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.236521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1d92e6-d406-43b8-a424-12bd2bb51b89-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm\" (UID: \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:43.236639 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.236550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393eee2a-9aa2-465a-b56c-e97fe676bb63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v\" (UID: \"393eee2a-9aa2-465a-b56c-e97fe676bb63\") " pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:43.236639 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.236622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvpdm\" (UniqueName: \"kubernetes.io/projected/9d1d92e6-d406-43b8-a424-12bd2bb51b89-kube-api-access-pvpdm\") pod \"payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm\" (UID: \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:43.263744 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.263715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:43.337840 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.337807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1d92e6-d406-43b8-a424-12bd2bb51b89-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm\" (UID: \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:43.338300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.337846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393eee2a-9aa2-465a-b56c-e97fe676bb63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v\" (UID: \"393eee2a-9aa2-465a-b56c-e97fe676bb63\") " pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:43.338300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.337865    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvpdm\" (UniqueName: \"kubernetes.io/projected/9d1d92e6-d406-43b8-a424-12bd2bb51b89-kube-api-access-pvpdm\") pod \"payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm\" (UID: \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:43.338300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.337930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7hp\" (UniqueName: \"kubernetes.io/projected/393eee2a-9aa2-465a-b56c-e97fe676bb63-kube-api-access-2l7hp\") pod \"payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v\" (UID: \"393eee2a-9aa2-465a-b56c-e97fe676bb63\") " pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:43.338658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.338635    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393eee2a-9aa2-465a-b56c-e97fe676bb63-plugins-config-volume\") pod \"payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v\" (UID: \"393eee2a-9aa2-465a-b56c-e97fe676bb63\") " pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:43.338658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.338650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1d92e6-d406-43b8-a424-12bd2bb51b89-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm\" (UID: \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:43.349672 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.349648    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l7hp\" (UniqueName: \"kubernetes.io/projected/393eee2a-9aa2-465a-b56c-e97fe676bb63-kube-api-access-2l7hp\") pod \"payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v\" (UID: \"393eee2a-9aa2-465a-b56c-e97fe676bb63\") " pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:43.350479 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.350455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvpdm\" (UniqueName: \"kubernetes.io/projected/9d1d92e6-d406-43b8-a424-12bd2bb51b89-kube-api-access-pvpdm\") pod \"payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm\" (UID: \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:43.396231 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.396204    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"]
Jul 27 21:18:43.398535 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:18:43.398510    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4a3f75_b3f8_4f38_a4cf_a624001d7214.slice/crio-eaa258d9d662c1f1e3fa426df6f8a75415ef6869ba0c39ab98dce70c5e2a051e WatchSource:0}: Error finding container eaa258d9d662c1f1e3fa426df6f8a75415ef6869ba0c39ab98dce70c5e2a051e: Status 404 returned error can't find the container with id eaa258d9d662c1f1e3fa426df6f8a75415ef6869ba0c39ab98dce70c5e2a051e
Jul 27 21:18:43.479425 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.479395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:43.509382 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.509351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:43.528746 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.528704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk" event={"ID":"4b4a3f75-b3f8-4f38-a4cf-a624001d7214","Type":"ContainerStarted","Data":"eaa258d9d662c1f1e3fa426df6f8a75415ef6869ba0c39ab98dce70c5e2a051e"}
Jul 27 21:18:43.633587 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.633550    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"]
Jul 27 21:18:43.637290 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:18:43.637250    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393eee2a_9aa2_465a_b56c_e97fe676bb63.slice/crio-db4795b5c94975d64b24779ed4abe970858ab387ff2efbcdb1bbd2da47f47cf5 WatchSource:0}: Error finding container db4795b5c94975d64b24779ed4abe970858ab387ff2efbcdb1bbd2da47f47cf5: Status 404 returned error can't find the container with id db4795b5c94975d64b24779ed4abe970858ab387ff2efbcdb1bbd2da47f47cf5
Jul 27 21:18:43.686820 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:43.686797    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"]
Jul 27 21:18:43.695276 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:18:43.695246    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1d92e6_d406_43b8_a424_12bd2bb51b89.slice/crio-a08ebe831ca17f41688284af28e0b181509eb2d78ffb0e467f8a3e31219c8677 WatchSource:0}: Error finding container a08ebe831ca17f41688284af28e0b181509eb2d78ffb0e467f8a3e31219c8677: Status 404 returned error can't find the container with id a08ebe831ca17f41688284af28e0b181509eb2d78ffb0e467f8a3e31219c8677
Jul 27 21:18:44.537626 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:44.537588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v" event={"ID":"393eee2a-9aa2-465a-b56c-e97fe676bb63","Type":"ContainerStarted","Data":"120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96"}
Jul 27 21:18:44.538222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:44.537648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v" event={"ID":"393eee2a-9aa2-465a-b56c-e97fe676bb63","Type":"ContainerStarted","Data":"db4795b5c94975d64b24779ed4abe970858ab387ff2efbcdb1bbd2da47f47cf5"}
Jul 27 21:18:44.538222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:44.537831    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:44.539549 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:44.539521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm" event={"ID":"9d1d92e6-d406-43b8-a424-12bd2bb51b89","Type":"ContainerStarted","Data":"c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7"}
Jul 27 21:18:44.539665 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:44.539572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm" event={"ID":"9d1d92e6-d406-43b8-a424-12bd2bb51b89","Type":"ContainerStarted","Data":"a08ebe831ca17f41688284af28e0b181509eb2d78ffb0e467f8a3e31219c8677"}
Jul 27 21:18:44.539778 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:44.539759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:18:44.553440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:44.553390    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v" podStartSLOduration=1.5533746430000002 podStartE2EDuration="1.553374643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:44.55295666 +0000 UTC m=+1641.918522508" watchObservedRunningTime="2026-07-27 21:18:44.553374643 +0000 UTC m=+1641.918940489"
Jul 27 21:18:44.568114 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:44.567904    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm" podStartSLOduration=1.567882961 podStartE2EDuration="1.567882961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:44.567531866 +0000 UTC m=+1641.933097714" watchObservedRunningTime="2026-07-27 21:18:44.567882961 +0000 UTC m=+1641.933448810"
Jul 27 21:18:46.551596 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:46.551560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk" event={"ID":"4b4a3f75-b3f8-4f38-a4cf-a624001d7214","Type":"ContainerStarted","Data":"fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b"}
Jul 27 21:18:46.552055 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:46.551693    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:46.570247 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:46.570201    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk" podStartSLOduration=2.342583985 podStartE2EDuration="4.570173878s" totalImagesPullingTime="2.227589893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:18:46.568146294 +0000 UTC m=+1643.933712142" watchObservedRunningTime="2026-07-27 21:18:46.570173878 +0000 UTC m=+1643.935739761"
Jul 27 21:18:48.447645 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:48.447607    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:18:52.562868 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:52.562830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:18:53.647473 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:53.647438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:18:55.547396 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:55.547350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:18:55.547793 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:18:55.547436    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:19:17.875995 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:17.875961    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"]
Jul 27 21:19:17.876542 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:17.876276    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk" podUID="4b4a3f75-b3f8-4f38-a4cf-a624001d7214" containerName="maas-api" containerID="cri-o://fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b" gracePeriod=30
Jul 27 21:19:17.922110 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:17.922070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:17.944494 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:17.944464    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"]
Jul 27 21:19:17.944704 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:17.944681    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v" podUID="393eee2a-9aa2-465a-b56c-e97fe676bb63" containerName="payload-processing" containerID="cri-o://120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96" gracePeriod=30
Jul 27 21:19:17.952586 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:17.952557    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"]
Jul 27 21:19:17.952841 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:17.952792    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm" podUID="9d1d92e6-d406-43b8-a424-12bd2bb51b89" containerName="payload-pre-processing" containerID="cri-o://c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7" gracePeriod=30
Jul 27 21:19:17.989123 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:17.989084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:18.065656 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.065621    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:18.244642 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.244617    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:19:18.247579 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.247552    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:19:18.250906 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.250886    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:19:18.370726 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.370687    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mdqh\" (UniqueName: \"kubernetes.io/projected/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-kube-api-access-2mdqh\") pod \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") "
Jul 27 21:19:18.370726 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.370735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls\") pod \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") "
Jul 27 21:19:18.370981 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.370781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-openshift-service-ca\") pod \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\" (UID: \"4b4a3f75-b3f8-4f38-a4cf-a624001d7214\") "
Jul 27 21:19:18.370981 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.370797    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393eee2a-9aa2-465a-b56c-e97fe676bb63-plugins-config-volume\") pod \"393eee2a-9aa2-465a-b56c-e97fe676bb63\" (UID: \"393eee2a-9aa2-465a-b56c-e97fe676bb63\") "
Jul 27 21:19:18.370981 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.370813    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvpdm\" (UniqueName: \"kubernetes.io/projected/9d1d92e6-d406-43b8-a424-12bd2bb51b89-kube-api-access-pvpdm\") pod \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\" (UID: \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\") "
Jul 27 21:19:18.370981 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.370832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l7hp\" (UniqueName: \"kubernetes.io/projected/393eee2a-9aa2-465a-b56c-e97fe676bb63-kube-api-access-2l7hp\") pod \"393eee2a-9aa2-465a-b56c-e97fe676bb63\" (UID: \"393eee2a-9aa2-465a-b56c-e97fe676bb63\") "
Jul 27 21:19:18.370981 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.370854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1d92e6-d406-43b8-a424-12bd2bb51b89-plugins-config-volume\") pod \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\" (UID: \"9d1d92e6-d406-43b8-a424-12bd2bb51b89\") "
Jul 27 21:19:18.371360 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.371303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393eee2a-9aa2-465a-b56c-e97fe676bb63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "393eee2a-9aa2-465a-b56c-e97fe676bb63" (UID: "393eee2a-9aa2-465a-b56c-e97fe676bb63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:19:18.371436 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.371366    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b4a3f75-b3f8-4f38-a4cf-a624001d7214" (UID: "4b4a3f75-b3f8-4f38-a4cf-a624001d7214"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:19:18.371475 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.371437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1d92e6-d406-43b8-a424-12bd2bb51b89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d1d92e6-d406-43b8-a424-12bd2bb51b89" (UID: "9d1d92e6-d406-43b8-a424-12bd2bb51b89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:19:18.373088 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.373065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1d92e6-d406-43b8-a424-12bd2bb51b89-kube-api-access-pvpdm" (OuterVolumeSpecName: "kube-api-access-pvpdm") pod "9d1d92e6-d406-43b8-a424-12bd2bb51b89" (UID: "9d1d92e6-d406-43b8-a424-12bd2bb51b89"). InnerVolumeSpecName "kube-api-access-pvpdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:19:18.373159 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.373123    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b4a3f75-b3f8-4f38-a4cf-a624001d7214" (UID: "4b4a3f75-b3f8-4f38-a4cf-a624001d7214"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:19:18.373602 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.373581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393eee2a-9aa2-465a-b56c-e97fe676bb63-kube-api-access-2l7hp" (OuterVolumeSpecName: "kube-api-access-2l7hp") pod "393eee2a-9aa2-465a-b56c-e97fe676bb63" (UID: "393eee2a-9aa2-465a-b56c-e97fe676bb63"). InnerVolumeSpecName "kube-api-access-2l7hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:19:18.373602 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.373587    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-kube-api-access-2mdqh" (OuterVolumeSpecName: "kube-api-access-2mdqh") pod "4b4a3f75-b3f8-4f38-a4cf-a624001d7214" (UID: "4b4a3f75-b3f8-4f38-a4cf-a624001d7214"). InnerVolumeSpecName "kube-api-access-2mdqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:19:18.472428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.472341    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:19:18.472428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.472370    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/393eee2a-9aa2-465a-b56c-e97fe676bb63-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:19:18.472428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.472381    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvpdm\" (UniqueName: \"kubernetes.io/projected/9d1d92e6-d406-43b8-a424-12bd2bb51b89-kube-api-access-pvpdm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:19:18.472428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.472391    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l7hp\" (UniqueName: \"kubernetes.io/projected/393eee2a-9aa2-465a-b56c-e97fe676bb63-kube-api-access-2l7hp\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:19:18.472428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.472399    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d1d92e6-d406-43b8-a424-12bd2bb51b89-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:19:18.472428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.472408    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mdqh\" (UniqueName: \"kubernetes.io/projected/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-kube-api-access-2mdqh\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:19:18.472428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.472417    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b4a3f75-b3f8-4f38-a4cf-a624001d7214-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:19:18.706980 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.706945    2577 generic.go:356] "Generic (PLEG): container finished" podID="393eee2a-9aa2-465a-b56c-e97fe676bb63" containerID="120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96" exitCode=0
Jul 27 21:19:18.707211 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.706998    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"
Jul 27 21:19:18.707211 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.707031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v" event={"ID":"393eee2a-9aa2-465a-b56c-e97fe676bb63","Type":"ContainerDied","Data":"120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96"}
Jul 27 21:19:18.707211 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.707079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v" event={"ID":"393eee2a-9aa2-465a-b56c-e97fe676bb63","Type":"ContainerDied","Data":"db4795b5c94975d64b24779ed4abe970858ab387ff2efbcdb1bbd2da47f47cf5"}
Jul 27 21:19:18.707211 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.707109    2577 scope.go:117] "RemoveContainer" containerID="120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96"
Jul 27 21:19:18.708339 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.708320    2577 generic.go:356] "Generic (PLEG): container finished" podID="4b4a3f75-b3f8-4f38-a4cf-a624001d7214" containerID="fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b" exitCode=0
Jul 27 21:19:18.708410 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.708368    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"
Jul 27 21:19:18.708410 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.708366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk" event={"ID":"4b4a3f75-b3f8-4f38-a4cf-a624001d7214","Type":"ContainerDied","Data":"fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b"}
Jul 27 21:19:18.708525 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.708425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk" event={"ID":"4b4a3f75-b3f8-4f38-a4cf-a624001d7214","Type":"ContainerDied","Data":"eaa258d9d662c1f1e3fa426df6f8a75415ef6869ba0c39ab98dce70c5e2a051e"}
Jul 27 21:19:18.710359 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.710323    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d1d92e6-d406-43b8-a424-12bd2bb51b89" containerID="c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7" exitCode=0
Jul 27 21:19:18.710482 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.710368    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"
Jul 27 21:19:18.710482 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.710389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm" event={"ID":"9d1d92e6-d406-43b8-a424-12bd2bb51b89","Type":"ContainerDied","Data":"c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7"}
Jul 27 21:19:18.710482 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.710430    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm" event={"ID":"9d1d92e6-d406-43b8-a424-12bd2bb51b89","Type":"ContainerDied","Data":"a08ebe831ca17f41688284af28e0b181509eb2d78ffb0e467f8a3e31219c8677"}
Jul 27 21:19:18.718665 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.718641    2577 scope.go:117] "RemoveContainer" containerID="120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96"
Jul 27 21:19:18.719006 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:19:18.718985    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96\": container with ID starting with 120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96 not found: ID does not exist" containerID="120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96"
Jul 27 21:19:18.719079 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.719014    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96"} err="failed to get container status \"120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96\": rpc error: code = NotFound desc = could not find container \"120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96\": container with ID starting with 120ff2bed56260a06924350fea429ec474e7621b694bf49c18091faf9aca1b96 not found: ID does not exist"
Jul 27 21:19:18.719079 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.719031    2577 scope.go:117] "RemoveContainer" containerID="fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b"
Jul 27 21:19:18.727668 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.727654    2577 scope.go:117] "RemoveContainer" containerID="fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b"
Jul 27 21:19:18.727924 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:19:18.727906    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b\": container with ID starting with fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b not found: ID does not exist" containerID="fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b"
Jul 27 21:19:18.727968 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.727932    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b"} err="failed to get container status \"fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b\": rpc error: code = NotFound desc = could not find container \"fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b\": container with ID starting with fffc959acb9400a330c0795a4063b07a667a59cc8112ee3c92a0231732c2932b not found: ID does not exist"
Jul 27 21:19:18.727968 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.727948    2577 scope.go:117] "RemoveContainer" containerID="c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7"
Jul 27 21:19:18.734074 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.734056    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"]
Jul 27 21:19:18.736549 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.736494    2577 scope.go:117] "RemoveContainer" containerID="c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7"
Jul 27 21:19:18.736641 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.736606    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20b824d1-cd9f5d9f7-gt87v"]
Jul 27 21:19:18.736824 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:19:18.736803    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7\": container with ID starting with c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7 not found: ID does not exist" containerID="c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7"
Jul 27 21:19:18.736883 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.736844    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7"} err="failed to get container status \"c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7\": rpc error: code = NotFound desc = could not find container \"c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7\": container with ID starting with c2cf3b29ceb01d2ad7775ac57cf055858766b24b6b76ecfd8672d076b90ffba7 not found: ID does not exist"
Jul 27 21:19:18.746636 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.746614    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"]
Jul 27 21:19:18.753888 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.753869    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20b824d1-79c9759976-kpbvm"]
Jul 27 21:19:18.763807 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.763784    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"]
Jul 27 21:19:18.767764 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:18.767744    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-20b824d1-d5cbd8d7d-7wmgk"]
Jul 27 21:19:19.185326 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:19.185291    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393eee2a-9aa2-465a-b56c-e97fe676bb63" path="/var/lib/kubelet/pods/393eee2a-9aa2-465a-b56c-e97fe676bb63/volumes"
Jul 27 21:19:19.185679 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:19.185633    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4a3f75-b3f8-4f38-a4cf-a624001d7214" path="/var/lib/kubelet/pods/4b4a3f75-b3f8-4f38-a4cf-a624001d7214/volumes"
Jul 27 21:19:19.185925 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:19.185913    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d1d92e6-d406-43b8-a424-12bd2bb51b89" path="/var/lib/kubelet/pods/9d1d92e6-d406-43b8-a424-12bd2bb51b89/volumes"
Jul 27 21:19:20.763523 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:20.763484    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:19:22.342314 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:22.342253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:22.500914 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:22.500875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:22.720483 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:22.720393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:22.936037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:22.935996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:23.263047 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:23.263018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:23.375203 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:23.375156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:25.364496 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.364459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:25.458146 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458108    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"]
Jul 27 21:19:25.458561 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458543    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393eee2a-9aa2-465a-b56c-e97fe676bb63" containerName="payload-processing"
Jul 27 21:19:25.458654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458562    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="393eee2a-9aa2-465a-b56c-e97fe676bb63" containerName="payload-processing"
Jul 27 21:19:25.458654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458596    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d1d92e6-d406-43b8-a424-12bd2bb51b89" containerName="payload-pre-processing"
Jul 27 21:19:25.458654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458606    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d1d92e6-d406-43b8-a424-12bd2bb51b89" containerName="payload-pre-processing"
Jul 27 21:19:25.458654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458634    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4a3f75-b3f8-4f38-a4cf-a624001d7214" containerName="maas-api"
Jul 27 21:19:25.458654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458642    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4a3f75-b3f8-4f38-a4cf-a624001d7214" containerName="maas-api"
Jul 27 21:19:25.458891 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458719    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="393eee2a-9aa2-465a-b56c-e97fe676bb63" containerName="payload-processing"
Jul 27 21:19:25.458891 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458734    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4a3f75-b3f8-4f38-a4cf-a624001d7214" containerName="maas-api"
Jul 27 21:19:25.458891 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.458744    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d1d92e6-d406-43b8-a424-12bd2bb51b89" containerName="payload-pre-processing"
Jul 27 21:19:25.463346 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.463310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:25.463500 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.463476    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.466504 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.466326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fc269dcd-serving-cert\""
Jul 27 21:19:25.467303 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.467285    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cgjq6\""
Jul 27 21:19:25.472811 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.467762    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:19:25.472811 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.468166    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:19:25.480616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.480589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"]
Jul 27 21:19:25.532757 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.532732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.532891 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.532766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjmrr\" (UniqueName: \"kubernetes.io/projected/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-kube-api-access-hjmrr\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.532891 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.532797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-openshift-service-ca\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.536119 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.536080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:25.633274 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.633205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjmrr\" (UniqueName: \"kubernetes.io/projected/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-kube-api-access-hjmrr\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.633274 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.633259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-openshift-service-ca\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.633422 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.633328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.633422 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:19:25.633415    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-serving-cert: secret "maas-api-e2e-ait-fc269dcd-serving-cert" not found
Jul 27 21:19:25.633488 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:19:25.633484    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls podName:7b2ba5a9-17fe-4d28-9c26-da7b46111e56 nodeName:}" failed. No retries permitted until 2026-07-27 21:19:26.133467005 +0000 UTC m=+1683.499032833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls") pod "maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd" (UID: "7b2ba5a9-17fe-4d28-9c26-da7b46111e56") : secret "maas-api-e2e-ait-fc269dcd-serving-cert" not found
Jul 27 21:19:25.633855 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.633836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-openshift-service-ca\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.649357 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.649332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjmrr\" (UniqueName: \"kubernetes.io/projected/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-kube-api-access-hjmrr\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:25.752021 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:25.751993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:26.137303 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.137268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:26.139769 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.139736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls\") pod \"maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:26.192458 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.192411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:26.292380 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.292345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:19:26.314835 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.314802    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"]
Jul 27 21:19:26.318194 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.318169    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:26.321394 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.321372    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fc269dcd\""
Jul 27 21:19:26.321394 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.321373    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fc269dcd-dockercfg-9gct8\""
Jul 27 21:19:26.329025 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.329003    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"]
Jul 27 21:19:26.345938 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.345907    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"]
Jul 27 21:19:26.349143 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.349128    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:26.361535 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.361512    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"]
Jul 27 21:19:26.384311 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.384266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:26.440340 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.440306    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03084d0b-5831-4960-8576-733a1b9f18d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99\" (UID: \"03084d0b-5831-4960-8576-733a1b9f18d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:26.440512 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.440470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010404df-6353-4f0a-89e0-b72273e46ad1-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj\" (UID: \"010404df-6353-4f0a-89e0-b72273e46ad1\") " pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:26.440635 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.440514    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr6f7\" (UniqueName: \"kubernetes.io/projected/03084d0b-5831-4960-8576-733a1b9f18d0-kube-api-access-nr6f7\") pod \"payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99\" (UID: \"03084d0b-5831-4960-8576-733a1b9f18d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:26.440703 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.440631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwv8f\" (UniqueName: \"kubernetes.io/projected/010404df-6353-4f0a-89e0-b72273e46ad1-kube-api-access-xwv8f\") pod \"payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj\" (UID: \"010404df-6353-4f0a-89e0-b72273e46ad1\") " pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:26.515311 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.515226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"]
Jul 27 21:19:26.517507 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:19:26.517479    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2ba5a9_17fe_4d28_9c26_da7b46111e56.slice/crio-f041105c7dc23eb843dfb71e4e810bd73ac0587a9a36c3278a53ef8803c3cb24 WatchSource:0}: Error finding container f041105c7dc23eb843dfb71e4e810bd73ac0587a9a36c3278a53ef8803c3cb24: Status 404 returned error can't find the container with id f041105c7dc23eb843dfb71e4e810bd73ac0587a9a36c3278a53ef8803c3cb24
Jul 27 21:19:26.541372 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.541336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010404df-6353-4f0a-89e0-b72273e46ad1-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj\" (UID: \"010404df-6353-4f0a-89e0-b72273e46ad1\") " pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:26.541372 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.541379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr6f7\" (UniqueName: \"kubernetes.io/projected/03084d0b-5831-4960-8576-733a1b9f18d0-kube-api-access-nr6f7\") pod \"payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99\" (UID: \"03084d0b-5831-4960-8576-733a1b9f18d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:26.541527 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.541418    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwv8f\" (UniqueName: \"kubernetes.io/projected/010404df-6353-4f0a-89e0-b72273e46ad1-kube-api-access-xwv8f\") pod \"payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj\" (UID: \"010404df-6353-4f0a-89e0-b72273e46ad1\") " pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:26.541527 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.541448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03084d0b-5831-4960-8576-733a1b9f18d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99\" (UID: \"03084d0b-5831-4960-8576-733a1b9f18d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:26.541988 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.541968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010404df-6353-4f0a-89e0-b72273e46ad1-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj\" (UID: \"010404df-6353-4f0a-89e0-b72273e46ad1\") " pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:26.542139 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.542119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03084d0b-5831-4960-8576-733a1b9f18d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99\" (UID: \"03084d0b-5831-4960-8576-733a1b9f18d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:26.553761 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.553734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwv8f\" (UniqueName: \"kubernetes.io/projected/010404df-6353-4f0a-89e0-b72273e46ad1-kube-api-access-xwv8f\") pod \"payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj\" (UID: \"010404df-6353-4f0a-89e0-b72273e46ad1\") " pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:26.553847 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.553731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr6f7\" (UniqueName: \"kubernetes.io/projected/03084d0b-5831-4960-8576-733a1b9f18d0-kube-api-access-nr6f7\") pod \"payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99\" (UID: \"03084d0b-5831-4960-8576-733a1b9f18d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:26.629409 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.629377    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:26.658736 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.658673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:26.754856 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.754827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd" event={"ID":"7b2ba5a9-17fe-4d28-9c26-da7b46111e56","Type":"ContainerStarted","Data":"f041105c7dc23eb843dfb71e4e810bd73ac0587a9a36c3278a53ef8803c3cb24"}
Jul 27 21:19:26.784898 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.784855    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"]
Jul 27 21:19:26.787913 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:19:26.787887    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03084d0b_5831_4960_8576_733a1b9f18d0.slice/crio-72fbaad439433a5b27f03299f141a089f55d2af585c369119dbf31a4126aaa4e WatchSource:0}: Error finding container 72fbaad439433a5b27f03299f141a089f55d2af585c369119dbf31a4126aaa4e: Status 404 returned error can't find the container with id 72fbaad439433a5b27f03299f141a089f55d2af585c369119dbf31a4126aaa4e
Jul 27 21:19:26.807875 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:26.807857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"]
Jul 27 21:19:26.810010 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:19:26.809989    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010404df_6353_4f0a_89e0_b72273e46ad1.slice/crio-b1f02a02b2117254503bfd7e71958aadadd2667a599b6c2df1befc87fb80a400 WatchSource:0}: Error finding container b1f02a02b2117254503bfd7e71958aadadd2667a599b6c2df1befc87fb80a400: Status 404 returned error can't find the container with id b1f02a02b2117254503bfd7e71958aadadd2667a599b6c2df1befc87fb80a400
Jul 27 21:19:27.761164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.761126    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd" event={"ID":"7b2ba5a9-17fe-4d28-9c26-da7b46111e56","Type":"ContainerStarted","Data":"fe6390381e71f253b9010caf1f09d13fb90e2347df76b14a1158c33c21e565e6"}
Jul 27 21:19:27.761687 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.761323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:27.762783 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.762750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj" event={"ID":"010404df-6353-4f0a-89e0-b72273e46ad1","Type":"ContainerStarted","Data":"370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5"}
Jul 27 21:19:27.762916 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.762807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj" event={"ID":"010404df-6353-4f0a-89e0-b72273e46ad1","Type":"ContainerStarted","Data":"b1f02a02b2117254503bfd7e71958aadadd2667a599b6c2df1befc87fb80a400"}
Jul 27 21:19:27.762916 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.762846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:27.764564 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.764540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99" event={"ID":"03084d0b-5831-4960-8576-733a1b9f18d0","Type":"ContainerStarted","Data":"1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976"}
Jul 27 21:19:27.764686 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.764582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99" event={"ID":"03084d0b-5831-4960-8576-733a1b9f18d0","Type":"ContainerStarted","Data":"72fbaad439433a5b27f03299f141a089f55d2af585c369119dbf31a4126aaa4e"}
Jul 27 21:19:27.764686 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.764625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:27.780085 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.780036    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd" podStartSLOduration=2.373443678 podStartE2EDuration="2.780020851s" totalImagesPullingTime="406.577173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:27.778345074 +0000 UTC m=+1685.143910920" watchObservedRunningTime="2026-07-27 21:19:27.780020851 +0000 UTC m=+1685.145586740"
Jul 27 21:19:27.795592 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.795538    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj" podStartSLOduration=1.795524128 podStartE2EDuration="1.795524128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:27.794339945 +0000 UTC m=+1685.159905795" watchObservedRunningTime="2026-07-27 21:19:27.795524128 +0000 UTC m=+1685.161089953"
Jul 27 21:19:27.811905 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:27.811860    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99" podStartSLOduration=1.811843445 podStartE2EDuration="1.811843445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:27.810501276 +0000 UTC m=+1685.176067126" watchObservedRunningTime="2026-07-27 21:19:27.811843445 +0000 UTC m=+1685.177409293"
Jul 27 21:19:31.756881 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:31.756847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:19:33.774315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:33.774287    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:19:38.771373 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:38.771342    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:19:38.771827 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:38.771460    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:19:51.082006 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.081968    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"]
Jul 27 21:19:51.086996 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.086979    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"
Jul 27 21:19:51.089617 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.089596    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fvvt\""
Jul 27 21:19:51.092882 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.092856    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"]
Jul 27 21:19:51.165053 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.165010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da536-9c67-4f72-925a-02031068ff9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn\" (UID: \"954da536-9c67-4f72-925a-02031068ff9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"
Jul 27 21:19:51.266401 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.266363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da536-9c67-4f72-925a-02031068ff9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn\" (UID: \"954da536-9c67-4f72-925a-02031068ff9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"
Jul 27 21:19:51.266980 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.266959    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da536-9c67-4f72-925a-02031068ff9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn\" (UID: \"954da536-9c67-4f72-925a-02031068ff9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"
Jul 27 21:19:51.399778 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.399700    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"
Jul 27 21:19:51.530996 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.530951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"]
Jul 27 21:19:51.533325 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:19:51.533298    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954da536_9c67_4f72_925a_02031068ff9b.slice/crio-e782d54daf38048b3b8bbdc6a9d413f35b388bf71fc00269c7db66acd3f3c505 WatchSource:0}: Error finding container e782d54daf38048b3b8bbdc6a9d413f35b388bf71fc00269c7db66acd3f3c505: Status 404 returned error can't find the container with id e782d54daf38048b3b8bbdc6a9d413f35b388bf71fc00269c7db66acd3f3c505
Jul 27 21:19:51.882785 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.882742    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn" event={"ID":"954da536-9c67-4f72-925a-02031068ff9b","Type":"ContainerStarted","Data":"1e5b598767a55566fc615d78150185f2929438cf17447dcbcdd2b6360a05c9e5"}
Jul 27 21:19:51.882966 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.882805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn" event={"ID":"954da536-9c67-4f72-925a-02031068ff9b","Type":"ContainerStarted","Data":"e782d54daf38048b3b8bbdc6a9d413f35b388bf71fc00269c7db66acd3f3c505"}
Jul 27 21:19:51.897461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:51.897415    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn" podStartSLOduration=0.897400414 podStartE2EDuration="897.400414ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:19:51.896451872 +0000 UTC m=+1709.262017719" watchObservedRunningTime="2026-07-27 21:19:51.897400414 +0000 UTC m=+1709.262966302"
Jul 27 21:19:52.889724 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:52.889687    2577 generic.go:356] "Generic (PLEG): container finished" podID="954da536-9c67-4f72-925a-02031068ff9b" containerID="1e5b598767a55566fc615d78150185f2929438cf17447dcbcdd2b6360a05c9e5" exitCode=0
Jul 27 21:19:52.890111 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:52.889730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn" event={"ID":"954da536-9c67-4f72-925a-02031068ff9b","Type":"ContainerDied","Data":"1e5b598767a55566fc615d78150185f2929438cf17447dcbcdd2b6360a05c9e5"}
Jul 27 21:19:54.026512 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:54.026484    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"
Jul 27 21:19:54.087464 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:54.087435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da536-9c67-4f72-925a-02031068ff9b-maas-api-service-ca\") pod \"954da536-9c67-4f72-925a-02031068ff9b\" (UID: \"954da536-9c67-4f72-925a-02031068ff9b\") "
Jul 27 21:19:54.087772 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:54.087752    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954da536-9c67-4f72-925a-02031068ff9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "954da536-9c67-4f72-925a-02031068ff9b" (UID: "954da536-9c67-4f72-925a-02031068ff9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:19:54.188589 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:54.188510    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954da536-9c67-4f72-925a-02031068ff9b-maas-api-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:19:54.899795 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:54.899764    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn" event={"ID":"954da536-9c67-4f72-925a-02031068ff9b","Type":"ContainerDied","Data":"e782d54daf38048b3b8bbdc6a9d413f35b388bf71fc00269c7db66acd3f3c505"}
Jul 27 21:19:54.899795 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:54.899789    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"
Jul 27 21:19:54.900015 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:19:54.899810    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e782d54daf38048b3b8bbdc6a9d413f35b388bf71fc00269c7db66acd3f3c505"
Jul 27 21:20:01.568923 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.568890    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"]
Jul 27 21:20:01.569328 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.569118    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd" podUID="7b2ba5a9-17fe-4d28-9c26-da7b46111e56" containerName="maas-api" containerID="cri-o://fe6390381e71f253b9010caf1f09d13fb90e2347df76b14a1158c33c21e565e6" gracePeriod=30
Jul 27 21:20:01.589366 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.589319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:01.611668 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.611640    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"]
Jul 27 21:20:01.611893 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.611871    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj" podUID="010404df-6353-4f0a-89e0-b72273e46ad1" containerName="payload-processing" containerID="cri-o://370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5" gracePeriod=30
Jul 27 21:20:01.619273 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.619249    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"]
Jul 27 21:20:01.619474 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.619453    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99" podUID="03084d0b-5831-4960-8576-733a1b9f18d0" containerName="payload-pre-processing" containerID="cri-o://1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976" gracePeriod=30
Jul 27 21:20:01.871531 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.871410    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:01.896790 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.896754    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:20:01.926386 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.926364    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:20:01.930609 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.930561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:01.937881 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.937847    2577 generic.go:356] "Generic (PLEG): container finished" podID="03084d0b-5831-4960-8576-733a1b9f18d0" containerID="1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976" exitCode=0
Jul 27 21:20:01.938004 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.937909    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"
Jul 27 21:20:01.938004 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.937946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99" event={"ID":"03084d0b-5831-4960-8576-733a1b9f18d0","Type":"ContainerDied","Data":"1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976"}
Jul 27 21:20:01.938004 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.938000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99" event={"ID":"03084d0b-5831-4960-8576-733a1b9f18d0","Type":"ContainerDied","Data":"72fbaad439433a5b27f03299f141a089f55d2af585c369119dbf31a4126aaa4e"}
Jul 27 21:20:01.938171 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.938025    2577 scope.go:117] "RemoveContainer" containerID="1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976"
Jul 27 21:20:01.939402 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.939382    2577 generic.go:356] "Generic (PLEG): container finished" podID="7b2ba5a9-17fe-4d28-9c26-da7b46111e56" containerID="fe6390381e71f253b9010caf1f09d13fb90e2347df76b14a1158c33c21e565e6" exitCode=0
Jul 27 21:20:01.939497 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.939459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd" event={"ID":"7b2ba5a9-17fe-4d28-9c26-da7b46111e56","Type":"ContainerDied","Data":"fe6390381e71f253b9010caf1f09d13fb90e2347df76b14a1158c33c21e565e6"}
Jul 27 21:20:01.940360 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.940331    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:20:01.940891 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.940870    2577 generic.go:356] "Generic (PLEG): container finished" podID="010404df-6353-4f0a-89e0-b72273e46ad1" containerID="370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5" exitCode=0
Jul 27 21:20:01.940995 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.940899    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"
Jul 27 21:20:01.940995 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.940929    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj" event={"ID":"010404df-6353-4f0a-89e0-b72273e46ad1","Type":"ContainerDied","Data":"370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5"}
Jul 27 21:20:01.940995 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.940976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj" event={"ID":"010404df-6353-4f0a-89e0-b72273e46ad1","Type":"ContainerDied","Data":"b1f02a02b2117254503bfd7e71958aadadd2667a599b6c2df1befc87fb80a400"}
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.949756    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr6f7\" (UniqueName: \"kubernetes.io/projected/03084d0b-5831-4960-8576-733a1b9f18d0-kube-api-access-nr6f7\") pod \"03084d0b-5831-4960-8576-733a1b9f18d0\" (UID: \"03084d0b-5831-4960-8576-733a1b9f18d0\") "
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.949827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwv8f\" (UniqueName: \"kubernetes.io/projected/010404df-6353-4f0a-89e0-b72273e46ad1-kube-api-access-xwv8f\") pod \"010404df-6353-4f0a-89e0-b72273e46ad1\" (UID: \"010404df-6353-4f0a-89e0-b72273e46ad1\") "
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.949912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03084d0b-5831-4960-8576-733a1b9f18d0-plugins-config-volume\") pod \"03084d0b-5831-4960-8576-733a1b9f18d0\" (UID: \"03084d0b-5831-4960-8576-733a1b9f18d0\") "
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.949961    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010404df-6353-4f0a-89e0-b72273e46ad1-plugins-config-volume\") pod \"010404df-6353-4f0a-89e0-b72273e46ad1\" (UID: \"010404df-6353-4f0a-89e0-b72273e46ad1\") "
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.950018    2577 scope.go:117] "RemoveContainer" containerID="1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976"
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:20:01.950855    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976\": container with ID starting with 1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976 not found: ID does not exist" containerID="1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976"
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.950895    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976"} err="failed to get container status \"1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976\": rpc error: code = NotFound desc = could not find container \"1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976\": container with ID starting with 1a34c40e8ad17f96a8bd8cb1a3242de50978d76713db4cc858d9d0e7fd964976 not found: ID does not exist"
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.950920    2577 scope.go:117] "RemoveContainer" containerID="370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5"
Jul 27 21:20:01.954256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.953032    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03084d0b-5831-4960-8576-733a1b9f18d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03084d0b-5831-4960-8576-733a1b9f18d0" (UID: "03084d0b-5831-4960-8576-733a1b9f18d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:01.954794 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.954416    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010404df-6353-4f0a-89e0-b72273e46ad1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "010404df-6353-4f0a-89e0-b72273e46ad1" (UID: "010404df-6353-4f0a-89e0-b72273e46ad1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:01.954794 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.954604    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03084d0b-5831-4960-8576-733a1b9f18d0-kube-api-access-nr6f7" (OuterVolumeSpecName: "kube-api-access-nr6f7") pod "03084d0b-5831-4960-8576-733a1b9f18d0" (UID: "03084d0b-5831-4960-8576-733a1b9f18d0"). InnerVolumeSpecName "kube-api-access-nr6f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:01.955946 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.955917    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010404df-6353-4f0a-89e0-b72273e46ad1-kube-api-access-xwv8f" (OuterVolumeSpecName: "kube-api-access-xwv8f") pod "010404df-6353-4f0a-89e0-b72273e46ad1" (UID: "010404df-6353-4f0a-89e0-b72273e46ad1"). InnerVolumeSpecName "kube-api-access-xwv8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:01.965087 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.965053    2577 scope.go:117] "RemoveContainer" containerID="370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5"
Jul 27 21:20:01.965482 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:20:01.965409    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5\": container with ID starting with 370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5 not found: ID does not exist" containerID="370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5"
Jul 27 21:20:01.965629 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:01.965449    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5"} err="failed to get container status \"370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5\": rpc error: code = NotFound desc = could not find container \"370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5\": container with ID starting with 370923ea5b39d4d81fc7a68efc236a7bdeb66e5b32f406614300db95ce204ba5 not found: ID does not exist"
Jul 27 21:20:02.050967 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.050927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls\") pod \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") "
Jul 27 21:20:02.051151 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.050988    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-openshift-service-ca\") pod \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") "
Jul 27 21:20:02.051151 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.051015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjmrr\" (UniqueName: \"kubernetes.io/projected/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-kube-api-access-hjmrr\") pod \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\" (UID: \"7b2ba5a9-17fe-4d28-9c26-da7b46111e56\") "
Jul 27 21:20:02.051315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.051250    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03084d0b-5831-4960-8576-733a1b9f18d0-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:02.051315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.051267    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010404df-6353-4f0a-89e0-b72273e46ad1-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:02.051315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.051280    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr6f7\" (UniqueName: \"kubernetes.io/projected/03084d0b-5831-4960-8576-733a1b9f18d0-kube-api-access-nr6f7\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:02.051315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.051293    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwv8f\" (UniqueName: \"kubernetes.io/projected/010404df-6353-4f0a-89e0-b72273e46ad1-kube-api-access-xwv8f\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:02.051517 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.051392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b2ba5a9-17fe-4d28-9c26-da7b46111e56" (UID: "7b2ba5a9-17fe-4d28-9c26-da7b46111e56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:02.053236 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.053217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b2ba5a9-17fe-4d28-9c26-da7b46111e56" (UID: "7b2ba5a9-17fe-4d28-9c26-da7b46111e56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:20:02.053341 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.053317    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-kube-api-access-hjmrr" (OuterVolumeSpecName: "kube-api-access-hjmrr") pod "7b2ba5a9-17fe-4d28-9c26-da7b46111e56" (UID: "7b2ba5a9-17fe-4d28-9c26-da7b46111e56"). InnerVolumeSpecName "kube-api-access-hjmrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:02.152418 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.152323    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:02.152418 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.152355    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:02.152418 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.152368    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjmrr\" (UniqueName: \"kubernetes.io/projected/7b2ba5a9-17fe-4d28-9c26-da7b46111e56-kube-api-access-hjmrr\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:02.259922 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.259890    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"]
Jul 27 21:20:02.261377 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.261342    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc269dcd-69d4f7f4f6-6tl99"]
Jul 27 21:20:02.270422 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.270399    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"]
Jul 27 21:20:02.274848 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.274827    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc269dcd-5cddc7fbd6-hnczj"]
Jul 27 21:20:02.948084 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.948000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd" event={"ID":"7b2ba5a9-17fe-4d28-9c26-da7b46111e56","Type":"ContainerDied","Data":"f041105c7dc23eb843dfb71e4e810bd73ac0587a9a36c3278a53ef8803c3cb24"}
Jul 27 21:20:02.948084 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.948047    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"
Jul 27 21:20:02.948084 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.948060    2577 scope.go:117] "RemoveContainer" containerID="fe6390381e71f253b9010caf1f09d13fb90e2347df76b14a1158c33c21e565e6"
Jul 27 21:20:02.969013 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.968986    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"]
Jul 27 21:20:02.972845 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:02.972824    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc269dcd-55d94f7745-5njxd"]
Jul 27 21:20:03.185604 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:03.185570    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="010404df-6353-4f0a-89e0-b72273e46ad1" path="/var/lib/kubelet/pods/010404df-6353-4f0a-89e0-b72273e46ad1/volumes"
Jul 27 21:20:03.185915 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:03.185903    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03084d0b-5831-4960-8576-733a1b9f18d0" path="/var/lib/kubelet/pods/03084d0b-5831-4960-8576-733a1b9f18d0/volumes"
Jul 27 21:20:03.186261 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:03.186248    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b2ba5a9-17fe-4d28-9c26-da7b46111e56" path="/var/lib/kubelet/pods/7b2ba5a9-17fe-4d28-9c26-da7b46111e56/volumes"
Jul 27 21:20:04.548806 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:04.548769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:20:06.161207 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:06.161079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:06.164952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:06.164882    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"]
Jul 27 21:20:06.166895 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:06.166874    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc269dcd-sspsn"]
Jul 27 21:20:07.184825 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:07.184794    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="954da536-9c67-4f72-925a-02031068ff9b" path="/var/lib/kubelet/pods/954da536-9c67-4f72-925a-02031068ff9b/volumes"
Jul 27 21:20:07.309582 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:07.309545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:07.523612 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:07.523565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:07.739675 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:07.739629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:08.129891 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:08.129857    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:08.950077 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:08.950040    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:10.636688 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.636652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:10.742813 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.742757    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:10.832396 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832361    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"]
Jul 27 21:20:10.832716 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832704    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03084d0b-5831-4960-8576-733a1b9f18d0" containerName="payload-pre-processing"
Jul 27 21:20:10.832782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832717    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="03084d0b-5831-4960-8576-733a1b9f18d0" containerName="payload-pre-processing"
Jul 27 21:20:10.832782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832724    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954da536-9c67-4f72-925a-02031068ff9b" containerName="revoke-keys"
Jul 27 21:20:10.832782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832730    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="954da536-9c67-4f72-925a-02031068ff9b" containerName="revoke-keys"
Jul 27 21:20:10.832782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832738    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="010404df-6353-4f0a-89e0-b72273e46ad1" containerName="payload-processing"
Jul 27 21:20:10.832782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832745    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="010404df-6353-4f0a-89e0-b72273e46ad1" containerName="payload-processing"
Jul 27 21:20:10.832782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832755    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2ba5a9-17fe-4d28-9c26-da7b46111e56" containerName="maas-api"
Jul 27 21:20:10.832782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832760    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2ba5a9-17fe-4d28-9c26-da7b46111e56" containerName="maas-api"
Jul 27 21:20:10.833147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832832    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2ba5a9-17fe-4d28-9c26-da7b46111e56" containerName="maas-api"
Jul 27 21:20:10.833147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832840    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="03084d0b-5831-4960-8576-733a1b9f18d0" containerName="payload-pre-processing"
Jul 27 21:20:10.833147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832845    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="954da536-9c67-4f72-925a-02031068ff9b" containerName="revoke-keys"
Jul 27 21:20:10.833147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.832852    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="010404df-6353-4f0a-89e0-b72273e46ad1" containerName="payload-processing"
Jul 27 21:20:10.837441 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.837413    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:10.839915 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.839893    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a9503936-serving-cert\""
Jul 27 21:20:10.840079 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.840022    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:20:10.840079 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.840057    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:20:10.841016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.841002    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cgjq6\""
Jul 27 21:20:10.846576 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.846552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"]
Jul 27 21:20:10.869242 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.869206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:10.928737 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.928643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-maas-api-tls\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:10.928737 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.928700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8fxn\" (UniqueName: \"kubernetes.io/projected/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-kube-api-access-w8fxn\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:10.928966 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:10.928748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-openshift-service-ca\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:11.029767 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.029733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-maas-api-tls\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:11.029963 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.029786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8fxn\" (UniqueName: \"kubernetes.io/projected/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-kube-api-access-w8fxn\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:11.029963 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.029831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-openshift-service-ca\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:11.030558 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.030539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-openshift-service-ca\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:11.032424 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.032409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-maas-api-tls\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:11.038312 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.038290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8fxn\" (UniqueName: \"kubernetes.io/projected/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-kube-api-access-w8fxn\") pod \"maas-api-e2e-derive-a9503936-64f5467c47-p46ks\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:11.070881 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.070847    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:11.150352 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.150311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:11.288925 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.288736    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"]
Jul 27 21:20:11.291175 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:20:11.291148    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc11b6c2_dcbb_40bf_a8cf_4389bed2b317.slice/crio-471a3e8103618b60cefc2182f2e6d4327bc27ef0506a9d4bcb458494c8623c0e WatchSource:0}: Error finding container 471a3e8103618b60cefc2182f2e6d4327bc27ef0506a9d4bcb458494c8623c0e: Status 404 returned error can't find the container with id 471a3e8103618b60cefc2182f2e6d4327bc27ef0506a9d4bcb458494c8623c0e
Jul 27 21:20:11.292603 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.292586    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:20:11.373621 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.373582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:11.473846 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.473723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:11.475719 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.475698    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"]
Jul 27 21:20:11.479943 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.479928    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:11.482971 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.482950    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a9503936-dockercfg-55hvq\""
Jul 27 21:20:11.483225 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.483209    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a9503936\""
Jul 27 21:20:11.489725 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.489700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"]
Jul 27 21:20:11.504669 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.504640    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"]
Jul 27 21:20:11.507811 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.507796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:11.518582 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.518559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"]
Jul 27 21:20:11.534225 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.534176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht7s8\" (UniqueName: \"kubernetes.io/projected/c2283fd7-136a-4f45-ae26-992f2bd01384-kube-api-access-ht7s8\") pod \"payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp\" (UID: \"c2283fd7-136a-4f45-ae26-992f2bd01384\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:11.534364 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.534256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2283fd7-136a-4f45-ae26-992f2bd01384-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp\" (UID: \"c2283fd7-136a-4f45-ae26-992f2bd01384\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:11.534364 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.534322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42618d11-e062-4506-8eec-3b141189c604-plugins-config-volume\") pod \"payload-processing-e2e-derive-a9503936-686469b47d-hbh7v\" (UID: \"42618d11-e062-4506-8eec-3b141189c604\") " pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:11.534364 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.534358    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrlh\" (UniqueName: \"kubernetes.io/projected/42618d11-e062-4506-8eec-3b141189c604-kube-api-access-7mrlh\") pod \"payload-processing-e2e-derive-a9503936-686469b47d-hbh7v\" (UID: \"42618d11-e062-4506-8eec-3b141189c604\") " pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:11.635554 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.635522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42618d11-e062-4506-8eec-3b141189c604-plugins-config-volume\") pod \"payload-processing-e2e-derive-a9503936-686469b47d-hbh7v\" (UID: \"42618d11-e062-4506-8eec-3b141189c604\") " pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:11.635749 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.635569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrlh\" (UniqueName: \"kubernetes.io/projected/42618d11-e062-4506-8eec-3b141189c604-kube-api-access-7mrlh\") pod \"payload-processing-e2e-derive-a9503936-686469b47d-hbh7v\" (UID: \"42618d11-e062-4506-8eec-3b141189c604\") " pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:11.635749 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.635612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht7s8\" (UniqueName: \"kubernetes.io/projected/c2283fd7-136a-4f45-ae26-992f2bd01384-kube-api-access-ht7s8\") pod \"payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp\" (UID: \"c2283fd7-136a-4f45-ae26-992f2bd01384\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:11.635749 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.635650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2283fd7-136a-4f45-ae26-992f2bd01384-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp\" (UID: \"c2283fd7-136a-4f45-ae26-992f2bd01384\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:11.636500 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.636477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42618d11-e062-4506-8eec-3b141189c604-plugins-config-volume\") pod \"payload-processing-e2e-derive-a9503936-686469b47d-hbh7v\" (UID: \"42618d11-e062-4506-8eec-3b141189c604\") " pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:11.636549 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.636477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2283fd7-136a-4f45-ae26-992f2bd01384-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp\" (UID: \"c2283fd7-136a-4f45-ae26-992f2bd01384\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:11.644884 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.644861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht7s8\" (UniqueName: \"kubernetes.io/projected/c2283fd7-136a-4f45-ae26-992f2bd01384-kube-api-access-ht7s8\") pod \"payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp\" (UID: \"c2283fd7-136a-4f45-ae26-992f2bd01384\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:11.645220 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.644865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrlh\" (UniqueName: \"kubernetes.io/projected/42618d11-e062-4506-8eec-3b141189c604-kube-api-access-7mrlh\") pod \"payload-processing-e2e-derive-a9503936-686469b47d-hbh7v\" (UID: \"42618d11-e062-4506-8eec-3b141189c604\") " pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:11.791786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.791748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:11.818623 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.818588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:11.958269 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.958043    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"]
Jul 27 21:20:11.960679 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:20:11.960649    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2283fd7_136a_4f45_ae26_992f2bd01384.slice/crio-b257347ed1964f5349466da7323bed638eab9b46516dca68f099061d76d3e6a4 WatchSource:0}: Error finding container b257347ed1964f5349466da7323bed638eab9b46516dca68f099061d76d3e6a4: Status 404 returned error can't find the container with id b257347ed1964f5349466da7323bed638eab9b46516dca68f099061d76d3e6a4
Jul 27 21:20:11.981281 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.981257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"]
Jul 27 21:20:11.983046 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:20:11.983025    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42618d11_e062_4506_8eec_3b141189c604.slice/crio-d45f330e2e4cd84eaa2b1a3ed4cabe3c45637d0699691e9e87731f89884554e5 WatchSource:0}: Error finding container d45f330e2e4cd84eaa2b1a3ed4cabe3c45637d0699691e9e87731f89884554e5: Status 404 returned error can't find the container with id d45f330e2e4cd84eaa2b1a3ed4cabe3c45637d0699691e9e87731f89884554e5
Jul 27 21:20:11.999239 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:11.999210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v" event={"ID":"42618d11-e062-4506-8eec-3b141189c604","Type":"ContainerStarted","Data":"d45f330e2e4cd84eaa2b1a3ed4cabe3c45637d0699691e9e87731f89884554e5"}
Jul 27 21:20:12.000487 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:12.000463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp" event={"ID":"c2283fd7-136a-4f45-ae26-992f2bd01384","Type":"ContainerStarted","Data":"b257347ed1964f5349466da7323bed638eab9b46516dca68f099061d76d3e6a4"}
Jul 27 21:20:12.002272 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:12.002249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks" event={"ID":"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317","Type":"ContainerStarted","Data":"f06ebfdc652d12b616b20b8bd6363142ae2eda6ecd25a1ef6620f98bfbeb8d63"}
Jul 27 21:20:12.002385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:12.002290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks" event={"ID":"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317","Type":"ContainerStarted","Data":"471a3e8103618b60cefc2182f2e6d4327bc27ef0506a9d4bcb458494c8623c0e"}
Jul 27 21:20:12.002458 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:12.002406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:12.019153 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:12.019103    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks" podStartSLOduration=1.462916811 podStartE2EDuration="2.01908489s" totalImagesPullingTime="556.168079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:12.018360462 +0000 UTC m=+1729.383926321" watchObservedRunningTime="2026-07-27 21:20:12.01908489 +0000 UTC m=+1729.384650741"
Jul 27 21:20:13.008340 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:13.008299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v" event={"ID":"42618d11-e062-4506-8eec-3b141189c604","Type":"ContainerStarted","Data":"e30caec7a1a9a38ee4432757abc78c3b5f6dd60c9cbaf0d5500e7d2666819614"}
Jul 27 21:20:13.008863 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:13.008403    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:13.009831 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:13.009810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp" event={"ID":"c2283fd7-136a-4f45-ae26-992f2bd01384","Type":"ContainerStarted","Data":"31ef92a484ea6763c70514240628366cb596a664bb6bdf6dbdbd2fe5a602d30d"}
Jul 27 21:20:13.009913 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:13.009872    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:13.025835 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:13.025788    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v" podStartSLOduration=2.025773096 podStartE2EDuration="2.025773096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:13.024713603 +0000 UTC m=+1730.390279449" watchObservedRunningTime="2026-07-27 21:20:13.025773096 +0000 UTC m=+1730.391338943"
Jul 27 21:20:13.040119 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:13.040077    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp" podStartSLOduration=2.04006328 podStartE2EDuration="2.04006328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:13.039808891 +0000 UTC m=+1730.405374754" watchObservedRunningTime="2026-07-27 21:20:13.04006328 +0000 UTC m=+1730.405629104"
Jul 27 21:20:16.049946 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:16.049916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:20:18.015834 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:18.015801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:24.016371 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:24.016339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:24.016767 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:24.016411    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:45.916142 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:45.916112    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"]
Jul 27 21:20:45.916562 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:45.916343    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks" podUID="dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" containerName="maas-api" containerID="cri-o://f06ebfdc652d12b616b20b8bd6363142ae2eda6ecd25a1ef6620f98bfbeb8d63" gracePeriod=30
Jul 27 21:20:45.942597 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:45.942573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:45.970126 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:45.970101    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"]
Jul 27 21:20:45.970351 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:45.970327    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v" podUID="42618d11-e062-4506-8eec-3b141189c604" containerName="payload-processing" containerID="cri-o://e30caec7a1a9a38ee4432757abc78c3b5f6dd60c9cbaf0d5500e7d2666819614" gracePeriod=30
Jul 27 21:20:45.978774 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:45.978749    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"]
Jul 27 21:20:45.979043 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:45.979018    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp" podUID="c2283fd7-136a-4f45-ae26-992f2bd01384" containerName="payload-pre-processing" containerID="cri-o://31ef92a484ea6763c70514240628366cb596a664bb6bdf6dbdbd2fe5a602d30d" gracePeriod=30
Jul 27 21:20:46.016806 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.016766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:46.086176 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.086143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:46.168916 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.168849    2577 generic.go:356] "Generic (PLEG): container finished" podID="dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" containerID="f06ebfdc652d12b616b20b8bd6363142ae2eda6ecd25a1ef6620f98bfbeb8d63" exitCode=0
Jul 27 21:20:46.169055 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.168922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks" event={"ID":"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317","Type":"ContainerDied","Data":"f06ebfdc652d12b616b20b8bd6363142ae2eda6ecd25a1ef6620f98bfbeb8d63"}
Jul 27 21:20:46.170377 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.170356    2577 generic.go:356] "Generic (PLEG): container finished" podID="42618d11-e062-4506-8eec-3b141189c604" containerID="e30caec7a1a9a38ee4432757abc78c3b5f6dd60c9cbaf0d5500e7d2666819614" exitCode=0
Jul 27 21:20:46.170455 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.170406    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v" event={"ID":"42618d11-e062-4506-8eec-3b141189c604","Type":"ContainerDied","Data":"e30caec7a1a9a38ee4432757abc78c3b5f6dd60c9cbaf0d5500e7d2666819614"}
Jul 27 21:20:46.171831 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.171813    2577 generic.go:356] "Generic (PLEG): container finished" podID="c2283fd7-136a-4f45-ae26-992f2bd01384" containerID="31ef92a484ea6763c70514240628366cb596a664bb6bdf6dbdbd2fe5a602d30d" exitCode=0
Jul 27 21:20:46.171902 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.171854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp" event={"ID":"c2283fd7-136a-4f45-ae26-992f2bd01384","Type":"ContainerDied","Data":"31ef92a484ea6763c70514240628366cb596a664bb6bdf6dbdbd2fe5a602d30d"}
Jul 27 21:20:46.240742 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.240717    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:46.251443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.251424    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:46.258865 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.258839    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:46.438063 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.437982    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8fxn\" (UniqueName: \"kubernetes.io/projected/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-kube-api-access-w8fxn\") pod \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") "
Jul 27 21:20:46.438063 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438045    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2283fd7-136a-4f45-ae26-992f2bd01384-plugins-config-volume\") pod \"c2283fd7-136a-4f45-ae26-992f2bd01384\" (UID: \"c2283fd7-136a-4f45-ae26-992f2bd01384\") "
Jul 27 21:20:46.438311 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mrlh\" (UniqueName: \"kubernetes.io/projected/42618d11-e062-4506-8eec-3b141189c604-kube-api-access-7mrlh\") pod \"42618d11-e062-4506-8eec-3b141189c604\" (UID: \"42618d11-e062-4506-8eec-3b141189c604\") "
Jul 27 21:20:46.438311 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438104    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht7s8\" (UniqueName: \"kubernetes.io/projected/c2283fd7-136a-4f45-ae26-992f2bd01384-kube-api-access-ht7s8\") pod \"c2283fd7-136a-4f45-ae26-992f2bd01384\" (UID: \"c2283fd7-136a-4f45-ae26-992f2bd01384\") "
Jul 27 21:20:46.438311 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-maas-api-tls\") pod \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") "
Jul 27 21:20:46.438311 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-openshift-service-ca\") pod \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\" (UID: \"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317\") "
Jul 27 21:20:46.438529 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42618d11-e062-4506-8eec-3b141189c604-plugins-config-volume\") pod \"42618d11-e062-4506-8eec-3b141189c604\" (UID: \"42618d11-e062-4506-8eec-3b141189c604\") "
Jul 27 21:20:46.438635 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" (UID: "dc11b6c2-dcbb-40bf-a8cf-4389bed2b317"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:46.438695 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2283fd7-136a-4f45-ae26-992f2bd01384-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2283fd7-136a-4f45-ae26-992f2bd01384" (UID: "c2283fd7-136a-4f45-ae26-992f2bd01384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:46.438695 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42618d11-e062-4506-8eec-3b141189c604-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42618d11-e062-4506-8eec-3b141189c604" (UID: "42618d11-e062-4506-8eec-3b141189c604"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:46.438911 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438866    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:46.438911 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438884    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42618d11-e062-4506-8eec-3b141189c604-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:46.438911 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.438895    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2283fd7-136a-4f45-ae26-992f2bd01384-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:46.440506 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.440477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" (UID: "dc11b6c2-dcbb-40bf-a8cf-4389bed2b317"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:20:46.440621 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.440502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-kube-api-access-w8fxn" (OuterVolumeSpecName: "kube-api-access-w8fxn") pod "dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" (UID: "dc11b6c2-dcbb-40bf-a8cf-4389bed2b317"). InnerVolumeSpecName "kube-api-access-w8fxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:46.440771 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.440744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42618d11-e062-4506-8eec-3b141189c604-kube-api-access-7mrlh" (OuterVolumeSpecName: "kube-api-access-7mrlh") pod "42618d11-e062-4506-8eec-3b141189c604" (UID: "42618d11-e062-4506-8eec-3b141189c604"). InnerVolumeSpecName "kube-api-access-7mrlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:46.440868 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.440794    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2283fd7-136a-4f45-ae26-992f2bd01384-kube-api-access-ht7s8" (OuterVolumeSpecName: "kube-api-access-ht7s8") pod "c2283fd7-136a-4f45-ae26-992f2bd01384" (UID: "c2283fd7-136a-4f45-ae26-992f2bd01384"). InnerVolumeSpecName "kube-api-access-ht7s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:46.540095 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.540062    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8fxn\" (UniqueName: \"kubernetes.io/projected/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-kube-api-access-w8fxn\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:46.540095 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.540090    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrlh\" (UniqueName: \"kubernetes.io/projected/42618d11-e062-4506-8eec-3b141189c604-kube-api-access-7mrlh\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:46.540308 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.540104    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht7s8\" (UniqueName: \"kubernetes.io/projected/c2283fd7-136a-4f45-ae26-992f2bd01384-kube-api-access-ht7s8\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:46.540308 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:46.540117    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:20:47.177610 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.177583    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"
Jul 27 21:20:47.178041 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.177628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp" event={"ID":"c2283fd7-136a-4f45-ae26-992f2bd01384","Type":"ContainerDied","Data":"b257347ed1964f5349466da7323bed638eab9b46516dca68f099061d76d3e6a4"}
Jul 27 21:20:47.178041 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.177697    2577 scope.go:117] "RemoveContainer" containerID="31ef92a484ea6763c70514240628366cb596a664bb6bdf6dbdbd2fe5a602d30d"
Jul 27 21:20:47.179172 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.179145    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"
Jul 27 21:20:47.180781 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.180723    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"
Jul 27 21:20:47.188426 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.186493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks" event={"ID":"dc11b6c2-dcbb-40bf-a8cf-4389bed2b317","Type":"ContainerDied","Data":"471a3e8103618b60cefc2182f2e6d4327bc27ef0506a9d4bcb458494c8623c0e"}
Jul 27 21:20:47.188426 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.186542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v" event={"ID":"42618d11-e062-4506-8eec-3b141189c604","Type":"ContainerDied","Data":"d45f330e2e4cd84eaa2b1a3ed4cabe3c45637d0699691e9e87731f89884554e5"}
Jul 27 21:20:47.191975 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.191961    2577 scope.go:117] "RemoveContainer" containerID="f06ebfdc652d12b616b20b8bd6363142ae2eda6ecd25a1ef6620f98bfbeb8d63"
Jul 27 21:20:47.202333 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.202315    2577 scope.go:117] "RemoveContainer" containerID="e30caec7a1a9a38ee4432757abc78c3b5f6dd60c9cbaf0d5500e7d2666819614"
Jul 27 21:20:47.211249 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.211228    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"]
Jul 27 21:20:47.212853 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.212834    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a9503936-64f5467c47-p46ks"]
Jul 27 21:20:47.223556 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.223538    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"]
Jul 27 21:20:47.226924 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.226903    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a9503936-686469b47d-hbh7v"]
Jul 27 21:20:47.236596 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.236579    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"]
Jul 27 21:20:47.240306 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:47.240288    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a9503936-7d59696cc6-lmrlp"]
Jul 27 21:20:48.874806 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:48.874771    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:20:49.184473 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:49.184392    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42618d11-e062-4506-8eec-3b141189c604" path="/var/lib/kubelet/pods/42618d11-e062-4506-8eec-3b141189c604/volumes"
Jul 27 21:20:49.184743 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:49.184730    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2283fd7-136a-4f45-ae26-992f2bd01384" path="/var/lib/kubelet/pods/c2283fd7-136a-4f45-ae26-992f2bd01384/volumes"
Jul 27 21:20:49.185058 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:49.185046    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" path="/var/lib/kubelet/pods/dc11b6c2-dcbb-40bf-a8cf-4389bed2b317/volumes"
Jul 27 21:20:50.534173 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:50.534141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:50.734757 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:50.734717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:50.953594 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:50.953512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.471936 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:51.471905    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.568932 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:51.568897    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:51.664599 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:51.664555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:53.035336 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:53.035307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:58.738314 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:58.738277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:58.882776 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:58.882728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:58.977766 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:58.977725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:59.078436 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:59.078349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:20:59.436486 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:20:59.436406    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:25.405659 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:25.405618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:32.228389 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:32.228342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:32.342341 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:32.342312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:32.428476 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:32.428441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:48.281893 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:48.281852    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:54.251129 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:54.251084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:54.315251 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:54.315211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:54.389493 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:54.389456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:21:54.486209 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:21:54.486144    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:04.954312 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:04.954268    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:05.719309 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:05.719264    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:12.295225 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:12.295172    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:12.404367 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:12.404327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:12.500209 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:12.500160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:12.595437 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:12.595331    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:15.577130 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:15.577095    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:22:19.685823 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:19.685785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:24.026303 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:24.026264    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:24.127962 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:24.127923    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:26.127911 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:26.127867    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:26.224368 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:26.224336    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:26.323378 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:26.323341    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:28.203012 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:28.202976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:31.657507 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:31.657471    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:22:34.048817 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:34.048782    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:34.166968 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:34.166922    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:34.710290 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:34.710257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:40.746248 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:40.746215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:40.767950 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:40.767918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:22:40.847256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:40.847223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:40.980492 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:40.980458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:41.272402 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:41.272371    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:41.426261 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:41.426223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:43.976485 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:43.976402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:44.095725 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.095696    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"]
Jul 27 21:22:44.096147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096130    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42618d11-e062-4506-8eec-3b141189c604" containerName="payload-processing"
Jul 27 21:22:44.096223 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096150    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42618d11-e062-4506-8eec-3b141189c604" containerName="payload-processing"
Jul 27 21:22:44.096223 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096162    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" containerName="maas-api"
Jul 27 21:22:44.096223 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096168    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" containerName="maas-api"
Jul 27 21:22:44.096223 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096203    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2283fd7-136a-4f45-ae26-992f2bd01384" containerName="payload-pre-processing"
Jul 27 21:22:44.096223 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096211    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2283fd7-136a-4f45-ae26-992f2bd01384" containerName="payload-pre-processing"
Jul 27 21:22:44.096404 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096264    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42618d11-e062-4506-8eec-3b141189c604" containerName="payload-processing"
Jul 27 21:22:44.096404 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096273    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc11b6c2-dcbb-40bf-a8cf-4389bed2b317" containerName="maas-api"
Jul 27 21:22:44.096404 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.096279    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2283fd7-136a-4f45-ae26-992f2bd01384" containerName="payload-pre-processing"
Jul 27 21:22:44.099507 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.099492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.102944 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.102923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cgjq6\""
Jul 27 21:22:44.103984 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.103963    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-61de35ca-serving-cert\""
Jul 27 21:22:44.104443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.104425    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:22:44.104558 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.104444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:22:44.113716 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.113683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"]
Jul 27 21:22:44.139590 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.139537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:44.166065 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.166031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257449c-9e58-422c-8c67-d2f171e3838e-openshift-service-ca\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.166288 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.166127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flfls\" (UniqueName: \"kubernetes.io/projected/d257449c-9e58-422c-8c67-d2f171e3838e-kube-api-access-flfls\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.166288 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.166160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.267232 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.267168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flfls\" (UniqueName: \"kubernetes.io/projected/d257449c-9e58-422c-8c67-d2f171e3838e-kube-api-access-flfls\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.267457 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.267246    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.267457 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.267395    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257449c-9e58-422c-8c67-d2f171e3838e-openshift-service-ca\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.267457 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:22:44.267419    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-serving-cert: secret "maas-api-e2e-mt-61de35ca-serving-cert" not found
Jul 27 21:22:44.267642 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:22:44.267489    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls podName:d257449c-9e58-422c-8c67-d2f171e3838e nodeName:}" failed. No retries permitted until 2026-07-27 21:22:44.767467219 +0000 UTC m=+1882.133033051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls") pod "maas-api-e2e-mt-61de35ca-8bc459db9-256cd" (UID: "d257449c-9e58-422c-8c67-d2f171e3838e") : secret "maas-api-e2e-mt-61de35ca-serving-cert" not found
Jul 27 21:22:44.268259 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.268240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257449c-9e58-422c-8c67-d2f171e3838e-openshift-service-ca\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.278347 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.278321    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:44.284535 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.284515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flfls\" (UniqueName: \"kubernetes.io/projected/d257449c-9e58-422c-8c67-d2f171e3838e-kube-api-access-flfls\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.457385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.457348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:44.702400 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.702303    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:44.772760 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.772723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.775351 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.775322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls\") pod \"maas-api-e2e-mt-61de35ca-8bc459db9-256cd\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:44.798442 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.798399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:44.804129 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.804105    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"]
Jul 27 21:22:44.807650 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.807634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:44.810315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.810295    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-61de35ca\""
Jul 27 21:22:44.810427 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.810295    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-61de35ca-dockercfg-vddfw\""
Jul 27 21:22:44.816851 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.816828    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"]
Jul 27 21:22:44.834242 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.834217    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"]
Jul 27 21:22:44.837387 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.837372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:44.849048 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.849026    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"]
Jul 27 21:22:44.873641 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.873610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbz4b\" (UniqueName: \"kubernetes.io/projected/497016ef-66d5-4e49-b9e7-358f3cff500a-kube-api-access-gbz4b\") pod \"payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx\" (UID: \"497016ef-66d5-4e49-b9e7-358f3cff500a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:44.873770 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.873689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497016ef-66d5-4e49-b9e7-358f3cff500a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx\" (UID: \"497016ef-66d5-4e49-b9e7-358f3cff500a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:44.974473 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.974377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb\" (UID: \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\") " pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:44.974473 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.974428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhrt\" (UniqueName: \"kubernetes.io/projected/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-kube-api-access-wdhrt\") pod \"payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb\" (UID: \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\") " pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:44.974473 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.974460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497016ef-66d5-4e49-b9e7-358f3cff500a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx\" (UID: \"497016ef-66d5-4e49-b9e7-358f3cff500a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:44.974686 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.974495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbz4b\" (UniqueName: \"kubernetes.io/projected/497016ef-66d5-4e49-b9e7-358f3cff500a-kube-api-access-gbz4b\") pod \"payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx\" (UID: \"497016ef-66d5-4e49-b9e7-358f3cff500a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:44.975093 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.975074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497016ef-66d5-4e49-b9e7-358f3cff500a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx\" (UID: \"497016ef-66d5-4e49-b9e7-358f3cff500a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:44.983113 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:44.983082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbz4b\" (UniqueName: \"kubernetes.io/projected/497016ef-66d5-4e49-b9e7-358f3cff500a-kube-api-access-gbz4b\") pod \"payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx\" (UID: \"497016ef-66d5-4e49-b9e7-358f3cff500a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:45.010532 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.010503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:45.075854 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.075740    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb\" (UID: \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\") " pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:45.075854 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.075793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhrt\" (UniqueName: \"kubernetes.io/projected/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-kube-api-access-wdhrt\") pod \"payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb\" (UID: \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\") " pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:45.077137 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.077085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-plugins-config-volume\") pod \"payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb\" (UID: \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\") " pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:45.095042 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.094990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhrt\" (UniqueName: \"kubernetes.io/projected/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-kube-api-access-wdhrt\") pod \"payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb\" (UID: \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\") " pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:45.119567 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.119536    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:45.146557 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.146506    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:45.174722 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.174633    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"]
Jul 27 21:22:45.175844 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:22:45.175800    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd257449c_9e58_422c_8c67_d2f171e3838e.slice/crio-0d6d48cfbd277829824948d72bf8a6767d91565d19c3b9b82c9af3bf38712ba4 WatchSource:0}: Error finding container 0d6d48cfbd277829824948d72bf8a6767d91565d19c3b9b82c9af3bf38712ba4: Status 404 returned error can't find the container with id 0d6d48cfbd277829824948d72bf8a6767d91565d19c3b9b82c9af3bf38712ba4
Jul 27 21:22:45.271068 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.271044    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"]
Jul 27 21:22:45.272877 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:22:45.272852    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497016ef_66d5_4e49_b9e7_358f3cff500a.slice/crio-c38a609b4473a3f6ac6e251a292c52869c0ec441e41639e08af6c700ccb32785 WatchSource:0}: Error finding container c38a609b4473a3f6ac6e251a292c52869c0ec441e41639e08af6c700ccb32785: Status 404 returned error can't find the container with id c38a609b4473a3f6ac6e251a292c52869c0ec441e41639e08af6c700ccb32785
Jul 27 21:22:45.292406 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.292384    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"]
Jul 27 21:22:45.294370 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:22:45.294348    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2a365e_bdc7_41e5_b1a2_335a09cd3fb3.slice/crio-16cf3d577db594b8ae4e19f5840de37c6a78cf0a8bee47e82125721fd1ae6c5b WatchSource:0}: Error finding container 16cf3d577db594b8ae4e19f5840de37c6a78cf0a8bee47e82125721fd1ae6c5b: Status 404 returned error can't find the container with id 16cf3d577db594b8ae4e19f5840de37c6a78cf0a8bee47e82125721fd1ae6c5b
Jul 27 21:22:45.734492 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.734455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd" event={"ID":"d257449c-9e58-422c-8c67-d2f171e3838e","Type":"ContainerStarted","Data":"ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b"}
Jul 27 21:22:45.734661 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.734541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd" event={"ID":"d257449c-9e58-422c-8c67-d2f171e3838e","Type":"ContainerStarted","Data":"0d6d48cfbd277829824948d72bf8a6767d91565d19c3b9b82c9af3bf38712ba4"}
Jul 27 21:22:45.734661 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.734573    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:45.735900 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.735868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb" event={"ID":"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3","Type":"ContainerStarted","Data":"81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0"}
Jul 27 21:22:45.736040 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.735927    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:45.736040 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.735952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb" event={"ID":"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3","Type":"ContainerStarted","Data":"16cf3d577db594b8ae4e19f5840de37c6a78cf0a8bee47e82125721fd1ae6c5b"}
Jul 27 21:22:45.737281 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.737251    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx" event={"ID":"497016ef-66d5-4e49-b9e7-358f3cff500a","Type":"ContainerStarted","Data":"0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b"}
Jul 27 21:22:45.737385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.737296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx" event={"ID":"497016ef-66d5-4e49-b9e7-358f3cff500a","Type":"ContainerStarted","Data":"c38a609b4473a3f6ac6e251a292c52869c0ec441e41639e08af6c700ccb32785"}
Jul 27 21:22:45.737385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.737380    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:45.754866 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.754821    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd" podStartSLOduration=1.320498814 podStartE2EDuration="1.754807216s" totalImagesPullingTime="434.308402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:22:45.752716651 +0000 UTC m=+1883.118282497" watchObservedRunningTime="2026-07-27 21:22:45.754807216 +0000 UTC m=+1883.120373062"
Jul 27 21:22:45.770796 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.770737    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb" podStartSLOduration=1.770719744 podStartE2EDuration="1.770719744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:22:45.769042574 +0000 UTC m=+1883.134608422" watchObservedRunningTime="2026-07-27 21:22:45.770719744 +0000 UTC m=+1883.136285770"
Jul 27 21:22:45.788992 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:45.788936    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx" podStartSLOduration=1.788917361 podStartE2EDuration="1.788917361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:22:45.787042926 +0000 UTC m=+1883.152608772" watchObservedRunningTime="2026-07-27 21:22:45.788917361 +0000 UTC m=+1883.154483209"
Jul 27 21:22:48.432811 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:48.432767    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:49.086625 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:49.086584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:49.183616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:49.183575    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:49.356466 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:49.356389    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:22:49.499239 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:49.499206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:51.748279 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:51.748238    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:22:55.475495 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:55.475447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:55.592938 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:55.592895    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:22:55.878552 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:55.878519    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"]
Jul 27 21:22:55.881785 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:55.881770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"
Jul 27 21:22:55.884489 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:55.884465    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fvvt\""
Jul 27 21:22:55.893740 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:55.893704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"]
Jul 27 21:22:55.974135 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:55.974084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e74770c-6c05-486d-8cc1-787f3368c105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61de35ca-6sszl\" (UID: \"0e74770c-6c05-486d-8cc1-787f3368c105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"
Jul 27 21:22:56.075342 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.075311    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e74770c-6c05-486d-8cc1-787f3368c105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61de35ca-6sszl\" (UID: \"0e74770c-6c05-486d-8cc1-787f3368c105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"
Jul 27 21:22:56.076111 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.076089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e74770c-6c05-486d-8cc1-787f3368c105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-61de35ca-6sszl\" (UID: \"0e74770c-6c05-486d-8cc1-787f3368c105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"
Jul 27 21:22:56.191845 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.191772    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"
Jul 27 21:22:56.325166 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.325139    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"]
Jul 27 21:22:56.327162 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:22:56.327135    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e74770c_6c05_486d_8cc1_787f3368c105.slice/crio-743e031c16b69fef94ec9a4b9b05a11d30f34b890a46d9156ee47f01d007490d WatchSource:0}: Error finding container 743e031c16b69fef94ec9a4b9b05a11d30f34b890a46d9156ee47f01d007490d: Status 404 returned error can't find the container with id 743e031c16b69fef94ec9a4b9b05a11d30f34b890a46d9156ee47f01d007490d
Jul 27 21:22:56.745017 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.744987    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:22:56.745466 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.745057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:22:56.792470 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.792437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl" event={"ID":"0e74770c-6c05-486d-8cc1-787f3368c105","Type":"ContainerStarted","Data":"91556179cd86b6cae4ccad1fd690b8b7e108f742941b0e5efc3bbd5fe55eeacc"}
Jul 27 21:22:56.792644 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.792491    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl" event={"ID":"0e74770c-6c05-486d-8cc1-787f3368c105","Type":"ContainerStarted","Data":"743e031c16b69fef94ec9a4b9b05a11d30f34b890a46d9156ee47f01d007490d"}
Jul 27 21:22:56.808850 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:56.808801    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl" podStartSLOduration=1.808784283 podStartE2EDuration="1.808784283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:22:56.807629937 +0000 UTC m=+1894.173195783" watchObservedRunningTime="2026-07-27 21:22:56.808784283 +0000 UTC m=+1894.174350129"
Jul 27 21:22:57.798560 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:57.798525    2577 generic.go:356] "Generic (PLEG): container finished" podID="0e74770c-6c05-486d-8cc1-787f3368c105" containerID="91556179cd86b6cae4ccad1fd690b8b7e108f742941b0e5efc3bbd5fe55eeacc" exitCode=0
Jul 27 21:22:57.799016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:57.798612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl" event={"ID":"0e74770c-6c05-486d-8cc1-787f3368c105","Type":"ContainerDied","Data":"91556179cd86b6cae4ccad1fd690b8b7e108f742941b0e5efc3bbd5fe55eeacc"}
Jul 27 21:22:58.941544 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:58.941518    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"
Jul 27 21:22:59.101860 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:59.101781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e74770c-6c05-486d-8cc1-787f3368c105-maas-api-service-ca\") pod \"0e74770c-6c05-486d-8cc1-787f3368c105\" (UID: \"0e74770c-6c05-486d-8cc1-787f3368c105\") "
Jul 27 21:22:59.102133 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:59.102110    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e74770c-6c05-486d-8cc1-787f3368c105-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e74770c-6c05-486d-8cc1-787f3368c105" (UID: "0e74770c-6c05-486d-8cc1-787f3368c105"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:59.203155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:59.203125    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e74770c-6c05-486d-8cc1-787f3368c105-maas-api-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:22:59.809571 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:59.809542    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"
Jul 27 21:22:59.809571 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:59.809554    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl" event={"ID":"0e74770c-6c05-486d-8cc1-787f3368c105","Type":"ContainerDied","Data":"743e031c16b69fef94ec9a4b9b05a11d30f34b890a46d9156ee47f01d007490d"}
Jul 27 21:22:59.809791 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:22:59.809596    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="743e031c16b69fef94ec9a4b9b05a11d30f34b890a46d9156ee47f01d007490d"
Jul 27 21:23:06.291784 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.291753    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"]
Jul 27 21:23:06.292297 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.291944    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd" podUID="d257449c-9e58-422c-8c67-d2f171e3838e" containerName="maas-api" containerID="cri-o://ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b" gracePeriod=30
Jul 27 21:23:06.321213 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.321159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:06.355345 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.355313    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"]
Jul 27 21:23:06.355580 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.355555    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb" podUID="5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3" containerName="payload-processing" containerID="cri-o://81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0" gracePeriod=30
Jul 27 21:23:06.362233 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.361624    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"]
Jul 27 21:23:06.362233 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.361823    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx" podUID="497016ef-66d5-4e49-b9e7-358f3cff500a" containerName="payload-pre-processing" containerID="cri-o://0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b" gracePeriod=30
Jul 27 21:23:06.390379 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.390345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:06.454914 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.454880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:06.634135 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.634110    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:23:06.636162 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.636146    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:23:06.651035 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.651013    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:23:06.664583 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.664559    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flfls\" (UniqueName: \"kubernetes.io/projected/d257449c-9e58-422c-8c67-d2f171e3838e-kube-api-access-flfls\") pod \"d257449c-9e58-422c-8c67-d2f171e3838e\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") "
Jul 27 21:23:06.664739 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.664643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbz4b\" (UniqueName: \"kubernetes.io/projected/497016ef-66d5-4e49-b9e7-358f3cff500a-kube-api-access-gbz4b\") pod \"497016ef-66d5-4e49-b9e7-358f3cff500a\" (UID: \"497016ef-66d5-4e49-b9e7-358f3cff500a\") "
Jul 27 21:23:06.664739 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.664689    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257449c-9e58-422c-8c67-d2f171e3838e-openshift-service-ca\") pod \"d257449c-9e58-422c-8c67-d2f171e3838e\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") "
Jul 27 21:23:06.664739 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.664720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-plugins-config-volume\") pod \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\" (UID: \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\") "
Jul 27 21:23:06.664932 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.664827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdhrt\" (UniqueName: \"kubernetes.io/projected/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-kube-api-access-wdhrt\") pod \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\" (UID: \"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3\") "
Jul 27 21:23:06.664988 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.664930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497016ef-66d5-4e49-b9e7-358f3cff500a-plugins-config-volume\") pod \"497016ef-66d5-4e49-b9e7-358f3cff500a\" (UID: \"497016ef-66d5-4e49-b9e7-358f3cff500a\") "
Jul 27 21:23:06.665049 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.664986    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls\") pod \"d257449c-9e58-422c-8c67-d2f171e3838e\" (UID: \"d257449c-9e58-422c-8c67-d2f171e3838e\") "
Jul 27 21:23:06.666150 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.665570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3" (UID: "5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:23:06.666150 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.666117    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497016ef-66d5-4e49-b9e7-358f3cff500a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "497016ef-66d5-4e49-b9e7-358f3cff500a" (UID: "497016ef-66d5-4e49-b9e7-358f3cff500a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:23:06.667863 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.667823    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d257449c-9e58-422c-8c67-d2f171e3838e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d257449c-9e58-422c-8c67-d2f171e3838e" (UID: "d257449c-9e58-422c-8c67-d2f171e3838e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:23:06.668332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.668301    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d257449c-9e58-422c-8c67-d2f171e3838e-kube-api-access-flfls" (OuterVolumeSpecName: "kube-api-access-flfls") pod "d257449c-9e58-422c-8c67-d2f171e3838e" (UID: "d257449c-9e58-422c-8c67-d2f171e3838e"). InnerVolumeSpecName "kube-api-access-flfls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:23:06.668485 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.668463    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d257449c-9e58-422c-8c67-d2f171e3838e" (UID: "d257449c-9e58-422c-8c67-d2f171e3838e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:23:06.668862 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.668838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497016ef-66d5-4e49-b9e7-358f3cff500a-kube-api-access-gbz4b" (OuterVolumeSpecName: "kube-api-access-gbz4b") pod "497016ef-66d5-4e49-b9e7-358f3cff500a" (UID: "497016ef-66d5-4e49-b9e7-358f3cff500a"). InnerVolumeSpecName "kube-api-access-gbz4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:23:06.669655 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.669627    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-kube-api-access-wdhrt" (OuterVolumeSpecName: "kube-api-access-wdhrt") pod "5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3" (UID: "5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3"). InnerVolumeSpecName "kube-api-access-wdhrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:23:06.765713 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.765677    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:23:06.765713 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.765707    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdhrt\" (UniqueName: \"kubernetes.io/projected/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3-kube-api-access-wdhrt\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:23:06.765713 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.765717    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497016ef-66d5-4e49-b9e7-358f3cff500a-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:23:06.765951 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.765729    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d257449c-9e58-422c-8c67-d2f171e3838e-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:23:06.765951 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.765738    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flfls\" (UniqueName: \"kubernetes.io/projected/d257449c-9e58-422c-8c67-d2f171e3838e-kube-api-access-flfls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:23:06.765951 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.765746    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbz4b\" (UniqueName: \"kubernetes.io/projected/497016ef-66d5-4e49-b9e7-358f3cff500a-kube-api-access-gbz4b\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:23:06.765951 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.765757    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d257449c-9e58-422c-8c67-d2f171e3838e-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:23:06.849196 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.849087    2577 generic.go:356] "Generic (PLEG): container finished" podID="d257449c-9e58-422c-8c67-d2f171e3838e" containerID="ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b" exitCode=0
Jul 27 21:23:06.849196 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.849132    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"
Jul 27 21:23:06.849443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.849178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd" event={"ID":"d257449c-9e58-422c-8c67-d2f171e3838e","Type":"ContainerDied","Data":"ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b"}
Jul 27 21:23:06.849443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.849256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd" event={"ID":"d257449c-9e58-422c-8c67-d2f171e3838e","Type":"ContainerDied","Data":"0d6d48cfbd277829824948d72bf8a6767d91565d19c3b9b82c9af3bf38712ba4"}
Jul 27 21:23:06.849443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.849286    2577 scope.go:117] "RemoveContainer" containerID="ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b"
Jul 27 21:23:06.850517 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.850496    2577 generic.go:356] "Generic (PLEG): container finished" podID="5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3" containerID="81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0" exitCode=0
Jul 27 21:23:06.850627 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.850540    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"
Jul 27 21:23:06.850627 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.850553    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb" event={"ID":"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3","Type":"ContainerDied","Data":"81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0"}
Jul 27 21:23:06.850627 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.850601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb" event={"ID":"5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3","Type":"ContainerDied","Data":"16cf3d577db594b8ae4e19f5840de37c6a78cf0a8bee47e82125721fd1ae6c5b"}
Jul 27 21:23:06.852100 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.852064    2577 generic.go:356] "Generic (PLEG): container finished" podID="497016ef-66d5-4e49-b9e7-358f3cff500a" containerID="0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b" exitCode=0
Jul 27 21:23:06.852100 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.852098    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"
Jul 27 21:23:06.852292 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.852148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx" event={"ID":"497016ef-66d5-4e49-b9e7-358f3cff500a","Type":"ContainerDied","Data":"0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b"}
Jul 27 21:23:06.852292 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.852210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx" event={"ID":"497016ef-66d5-4e49-b9e7-358f3cff500a","Type":"ContainerDied","Data":"c38a609b4473a3f6ac6e251a292c52869c0ec441e41639e08af6c700ccb32785"}
Jul 27 21:23:06.859765 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.859744    2577 scope.go:117] "RemoveContainer" containerID="ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b"
Jul 27 21:23:06.860024 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:23:06.859998    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b\": container with ID starting with ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b not found: ID does not exist" containerID="ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b"
Jul 27 21:23:06.860086 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.860035    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b"} err="failed to get container status \"ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b\": rpc error: code = NotFound desc = could not find container \"ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b\": container with ID starting with ac3db0faa468106f5e0224ece9d8cb6598123eefd78ca054e4dd2c9ecf31b42b not found: ID does not exist"
Jul 27 21:23:06.860086 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.860057    2577 scope.go:117] "RemoveContainer" containerID="81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0"
Jul 27 21:23:06.868441 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.868423    2577 scope.go:117] "RemoveContainer" containerID="81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0"
Jul 27 21:23:06.868683 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:23:06.868667    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0\": container with ID starting with 81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0 not found: ID does not exist" containerID="81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0"
Jul 27 21:23:06.868741 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.868689    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0"} err="failed to get container status \"81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0\": rpc error: code = NotFound desc = could not find container \"81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0\": container with ID starting with 81ef77de856b97990d25c818b0c2c6dd6120f5f1d1d0d01703045095e51474e0 not found: ID does not exist"
Jul 27 21:23:06.868741 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.868703    2577 scope.go:117] "RemoveContainer" containerID="0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b"
Jul 27 21:23:06.871836 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.871815    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"]
Jul 27 21:23:06.875091 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.875071    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-61de35ca-8bc459db9-256cd"]
Jul 27 21:23:06.876726 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.876710    2577 scope.go:117] "RemoveContainer" containerID="0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b"
Jul 27 21:23:06.876959 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:23:06.876942    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b\": container with ID starting with 0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b not found: ID does not exist" containerID="0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b"
Jul 27 21:23:06.877020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.876968    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b"} err="failed to get container status \"0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b\": rpc error: code = NotFound desc = could not find container \"0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b\": container with ID starting with 0ca3fc2ad63089914c1c684befeb624e0adaf6be3593b34aa9f75f7fe046af5b not found: ID does not exist"
Jul 27 21:23:06.884564 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.884544    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"]
Jul 27 21:23:06.888268 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.888249    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-61de35ca-59c6898cbd-6plmx"]
Jul 27 21:23:06.903967 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.899442    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"]
Jul 27 21:23:06.906029 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:06.906003    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-61de35ca-6b6dbf9766-724mb"]
Jul 27 21:23:07.185779 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:07.185698    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="497016ef-66d5-4e49-b9e7-358f3cff500a" path="/var/lib/kubelet/pods/497016ef-66d5-4e49-b9e7-358f3cff500a/volumes"
Jul 27 21:23:07.186043 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:07.186030    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3" path="/var/lib/kubelet/pods/5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3/volumes"
Jul 27 21:23:07.186393 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:07.186381    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d257449c-9e58-422c-8c67-d2f171e3838e" path="/var/lib/kubelet/pods/d257449c-9e58-422c-8c67-d2f171e3838e/volumes"
Jul 27 21:23:09.343653 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:09.343616    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:23:11.015068 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:11.015032    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"]
Jul 27 21:23:11.018284 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:11.018258    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-61de35ca-6sszl"]
Jul 27 21:23:11.018956 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:11.018933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:11.185060 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:11.185026    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e74770c-6c05-486d-8cc1-787f3368c105" path="/var/lib/kubelet/pods/0e74770c-6c05-486d-8cc1-787f3368c105/volumes"
Jul 27 21:23:11.384669 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:11.384577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:17.251701 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:17.251662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:17.353986 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:17.353926    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:17.516139 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:17.516046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:17.718662 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:17.718625    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:17.852156 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:17.852050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:17.946076 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:17.946040    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:18.046855 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:18.046821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:23.943424 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:23.943384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:24.063346 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:24.063307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:24.167006 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:24.166971    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:24.328034 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:24.327997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:30.176317 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:30.176236    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:30.277044 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:30.276998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:37.138008 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:37.137955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:38.163049 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:38.163021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:40.259363 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.259332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:40.343492 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.343456    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"]
Jul 27 21:23:40.344043 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344020    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e74770c-6c05-486d-8cc1-787f3368c105" containerName="revoke-keys"
Jul 27 21:23:40.344043 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344045    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e74770c-6c05-486d-8cc1-787f3368c105" containerName="revoke-keys"
Jul 27 21:23:40.344277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344067    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497016ef-66d5-4e49-b9e7-358f3cff500a" containerName="payload-pre-processing"
Jul 27 21:23:40.344277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344071    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:40.344277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344101    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="497016ef-66d5-4e49-b9e7-358f3cff500a" containerName="payload-pre-processing"
Jul 27 21:23:40.344277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344119    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3" containerName="payload-processing"
Jul 27 21:23:40.344277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344128    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3" containerName="payload-processing"
Jul 27 21:23:40.344277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344154    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d257449c-9e58-422c-8c67-d2f171e3838e" containerName="maas-api"
Jul 27 21:23:40.344277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344163    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d257449c-9e58-422c-8c67-d2f171e3838e" containerName="maas-api"
Jul 27 21:23:40.344494 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344377    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="497016ef-66d5-4e49-b9e7-358f3cff500a" containerName="payload-pre-processing"
Jul 27 21:23:40.344494 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344396    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2a365e-bdc7-41e5-b1a2-335a09cd3fb3" containerName="payload-processing"
Jul 27 21:23:40.344494 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344407    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d257449c-9e58-422c-8c67-d2f171e3838e" containerName="maas-api"
Jul 27 21:23:40.344494 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.344414    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e74770c-6c05-486d-8cc1-787f3368c105" containerName="revoke-keys"
Jul 27 21:23:40.347798 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.347782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.351809 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.351790    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:23:40.352313 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.352289    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-316c1e32-serving-cert\""
Jul 27 21:23:40.352313 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.352302    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cgjq6\""
Jul 27 21:23:40.352501 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.352301    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:23:40.360384 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.360351    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"]
Jul 27 21:23:40.422761 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.422728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:40.458864 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.458833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fc325e-a764-46be-955b-068a23b77aab-openshift-service-ca\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.459061 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.458877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/12fc325e-a764-46be-955b-068a23b77aab-kube-api-access-qn92l\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.459061 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.458952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fc325e-a764-46be-955b-068a23b77aab-maas-api-tls\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.517707 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.517619    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:40.560393 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.560361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fc325e-a764-46be-955b-068a23b77aab-openshift-service-ca\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.560576 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.560398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/12fc325e-a764-46be-955b-068a23b77aab-kube-api-access-qn92l\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.560576 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.560425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fc325e-a764-46be-955b-068a23b77aab-maas-api-tls\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.561041 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.561016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fc325e-a764-46be-955b-068a23b77aab-openshift-service-ca\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.563003 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.562986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fc325e-a764-46be-955b-068a23b77aab-maas-api-tls\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.574986 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.574964    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/12fc325e-a764-46be-955b-068a23b77aab-kube-api-access-qn92l\") pod \"maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.658537 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.658497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:40.800761 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:40.800735    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"]
Jul 27 21:23:40.803575 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:23:40.803541    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fc325e_a764_46be_955b_068a23b77aab.slice/crio-b7121f750c65d501d498e1d57fed5f6f23d4e8dc780ebb151004b21416d26e49 WatchSource:0}: Error finding container b7121f750c65d501d498e1d57fed5f6f23d4e8dc780ebb151004b21416d26e49: Status 404 returned error can't find the container with id b7121f750c65d501d498e1d57fed5f6f23d4e8dc780ebb151004b21416d26e49
Jul 27 21:23:41.018704 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.018669    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m" event={"ID":"12fc325e-a764-46be-955b-068a23b77aab","Type":"ContainerStarted","Data":"b7121f750c65d501d498e1d57fed5f6f23d4e8dc780ebb151004b21416d26e49"}
Jul 27 21:23:41.021555 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.021528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:41.124468 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.124356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:41.124909 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.124889    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"]
Jul 27 21:23:41.129383 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.129368    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:41.131778 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.131753    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-316c1e32\""
Jul 27 21:23:41.131904 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.131759    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-316c1e32-dockercfg-vv4j9\""
Jul 27 21:23:41.143873 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.140636    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"]
Jul 27 21:23:41.147820 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.147791    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"]
Jul 27 21:23:41.151384 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.151348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:41.161909 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.161881    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"]
Jul 27 21:23:41.268324 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.268291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t57n\" (UniqueName: \"kubernetes.io/projected/b9fccb0d-37ab-403a-912c-d52f68f31b71-kube-api-access-7t57n\") pod \"payload-processing-e2e-mt-316c1e32-6947444d58-hq94k\" (UID: \"b9fccb0d-37ab-403a-912c-d52f68f31b71\") " pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:41.268670 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.268355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47zl7\" (UniqueName: \"kubernetes.io/projected/15d25fe2-d44d-4018-a420-3b9c76f868ad-kube-api-access-47zl7\") pod \"payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6\" (UID: \"15d25fe2-d44d-4018-a420-3b9c76f868ad\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:41.268670 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.268400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d25fe2-d44d-4018-a420-3b9c76f868ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6\" (UID: \"15d25fe2-d44d-4018-a420-3b9c76f868ad\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:41.268670 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.268486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fccb0d-37ab-403a-912c-d52f68f31b71-plugins-config-volume\") pod \"payload-processing-e2e-mt-316c1e32-6947444d58-hq94k\" (UID: \"b9fccb0d-37ab-403a-912c-d52f68f31b71\") " pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:41.369384 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.369347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47zl7\" (UniqueName: \"kubernetes.io/projected/15d25fe2-d44d-4018-a420-3b9c76f868ad-kube-api-access-47zl7\") pod \"payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6\" (UID: \"15d25fe2-d44d-4018-a420-3b9c76f868ad\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:41.369384 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.369389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d25fe2-d44d-4018-a420-3b9c76f868ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6\" (UID: \"15d25fe2-d44d-4018-a420-3b9c76f868ad\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:41.369660 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.369426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fccb0d-37ab-403a-912c-d52f68f31b71-plugins-config-volume\") pod \"payload-processing-e2e-mt-316c1e32-6947444d58-hq94k\" (UID: \"b9fccb0d-37ab-403a-912c-d52f68f31b71\") " pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:41.369660 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.369522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t57n\" (UniqueName: \"kubernetes.io/projected/b9fccb0d-37ab-403a-912c-d52f68f31b71-kube-api-access-7t57n\") pod \"payload-processing-e2e-mt-316c1e32-6947444d58-hq94k\" (UID: \"b9fccb0d-37ab-403a-912c-d52f68f31b71\") " pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:41.370210 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.370161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fccb0d-37ab-403a-912c-d52f68f31b71-plugins-config-volume\") pod \"payload-processing-e2e-mt-316c1e32-6947444d58-hq94k\" (UID: \"b9fccb0d-37ab-403a-912c-d52f68f31b71\") " pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:41.370279 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.370166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d25fe2-d44d-4018-a420-3b9c76f868ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6\" (UID: \"15d25fe2-d44d-4018-a420-3b9c76f868ad\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:41.379082 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.379020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t57n\" (UniqueName: \"kubernetes.io/projected/b9fccb0d-37ab-403a-912c-d52f68f31b71-kube-api-access-7t57n\") pod \"payload-processing-e2e-mt-316c1e32-6947444d58-hq94k\" (UID: \"b9fccb0d-37ab-403a-912c-d52f68f31b71\") " pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:41.379082 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.379020    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47zl7\" (UniqueName: \"kubernetes.io/projected/15d25fe2-d44d-4018-a420-3b9c76f868ad-kube-api-access-47zl7\") pod \"payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6\" (UID: \"15d25fe2-d44d-4018-a420-3b9c76f868ad\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:41.445993 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.445950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:41.462365 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.462335    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:41.614628 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.614564    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"]
Jul 27 21:23:41.617795 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:23:41.617763    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d25fe2_d44d_4018_a420_3b9c76f868ad.slice/crio-9f33a165bdad4c3f2e820482506392009f567813f0ec9dba0e85258ac2f0afec WatchSource:0}: Error finding container 9f33a165bdad4c3f2e820482506392009f567813f0ec9dba0e85258ac2f0afec: Status 404 returned error can't find the container with id 9f33a165bdad4c3f2e820482506392009f567813f0ec9dba0e85258ac2f0afec
Jul 27 21:23:41.644033 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:41.644011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"]
Jul 27 21:23:41.658422 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:23:41.658384    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fccb0d_37ab_403a_912c_d52f68f31b71.slice/crio-39d8085c5872582e4fd2864592d50f38c8c814d43dc0b562c115da63e86080f0 WatchSource:0}: Error finding container 39d8085c5872582e4fd2864592d50f38c8c814d43dc0b562c115da63e86080f0: Status 404 returned error can't find the container with id 39d8085c5872582e4fd2864592d50f38c8c814d43dc0b562c115da63e86080f0
Jul 27 21:23:42.024827 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.024789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6" event={"ID":"15d25fe2-d44d-4018-a420-3b9c76f868ad","Type":"ContainerStarted","Data":"cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551"}
Jul 27 21:23:42.025011 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.024855    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:42.025011 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.024884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6" event={"ID":"15d25fe2-d44d-4018-a420-3b9c76f868ad","Type":"ContainerStarted","Data":"9f33a165bdad4c3f2e820482506392009f567813f0ec9dba0e85258ac2f0afec"}
Jul 27 21:23:42.026302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.026275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k" event={"ID":"b9fccb0d-37ab-403a-912c-d52f68f31b71","Type":"ContainerStarted","Data":"10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33"}
Jul 27 21:23:42.026418 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.026314    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k" event={"ID":"b9fccb0d-37ab-403a-912c-d52f68f31b71","Type":"ContainerStarted","Data":"39d8085c5872582e4fd2864592d50f38c8c814d43dc0b562c115da63e86080f0"}
Jul 27 21:23:42.026418 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.026365    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:42.027732 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.027705    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m" event={"ID":"12fc325e-a764-46be-955b-068a23b77aab","Type":"ContainerStarted","Data":"6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab"}
Jul 27 21:23:42.027834 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.027759    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:42.041945 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.041893    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6" podStartSLOduration=1.041876812 podStartE2EDuration="1.041876812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:42.040305194 +0000 UTC m=+1939.405871043" watchObservedRunningTime="2026-07-27 21:23:42.041876812 +0000 UTC m=+1939.407442637"
Jul 27 21:23:42.059277 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.059227    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m" podStartSLOduration=1.6334892380000001 podStartE2EDuration="2.059206803s" totalImagesPullingTime="425.717565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:42.057109423 +0000 UTC m=+1939.422675270" watchObservedRunningTime="2026-07-27 21:23:42.059206803 +0000 UTC m=+1939.424772651"
Jul 27 21:23:42.075565 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:42.075511    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k" podStartSLOduration=1.075494672 podStartE2EDuration="1.075494672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:42.073718293 +0000 UTC m=+1939.439284142" watchObservedRunningTime="2026-07-27 21:23:42.075494672 +0000 UTC m=+1939.441060518"
Jul 27 21:23:44.756626 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:44.756596    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:44.861352 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:44.861322    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:23:48.039325 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:48.039286    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:23:53.036118 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:53.036081    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:23:53.036510 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:53.036167    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:23:55.827825 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.827793    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"]
Jul 27 21:23:55.831212 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.831178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.834824 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.834801    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-316c1e32\"/\"e2e-lifecycle-model-316c1e32-kserve-self-signed-certs\""
Jul 27 21:23:55.834940 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.834853    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-316c1e32\"/\"openshift-service-ca.crt\""
Jul 27 21:23:55.834940 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.834864    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-316c1e32\"/\"default-dockercfg-cdb66\""
Jul 27 21:23:55.834940 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.834905    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-316c1e32\"/\"kube-root-ca.crt\""
Jul 27 21:23:55.840645 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.840626    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"]
Jul 27 21:23:55.858397 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.858366    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:23:55.896006 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.895974    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-dshm\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.896142 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.896012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-home\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.896142 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.896030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-model-cache\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.896142 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.896044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-tmp-dir\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.896142 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.896119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556d9899-35a8-4b45-b682-2395bdedd2cc-tls-certs\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.896332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.896144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cn9q\" (UniqueName: \"kubernetes.io/projected/556d9899-35a8-4b45-b682-2395bdedd2cc-kube-api-access-8cn9q\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997073 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-home\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997293 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-model-cache\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997293 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-tmp-dir\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997293 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556d9899-35a8-4b45-b682-2395bdedd2cc-tls-certs\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997293 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cn9q\" (UniqueName: \"kubernetes.io/projected/556d9899-35a8-4b45-b682-2395bdedd2cc-kube-api-access-8cn9q\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997525 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-dshm\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997587 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-home\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997587 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-model-cache\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.997693 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.997594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-tmp-dir\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.999614 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.999594    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-dshm\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:55.999767 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:55.999749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556d9899-35a8-4b45-b682-2395bdedd2cc-tls-certs\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:56.005893 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:56.005869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cn9q\" (UniqueName: \"kubernetes.io/projected/556d9899-35a8-4b45-b682-2395bdedd2cc-kube-api-access-8cn9q\") pod \"e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") " pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:56.141169 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:56.141085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:23:56.273152 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:56.273123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"]
Jul 27 21:23:56.274967 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:23:56.274936    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod556d9899_35a8_4b45_b682_2395bdedd2cc.slice/crio-131362ced33675196f38713b54694d8c2db04062e8b42a01e7655cdc039b05db WatchSource:0}: Error finding container 131362ced33675196f38713b54694d8c2db04062e8b42a01e7655cdc039b05db: Status 404 returned error can't find the container with id 131362ced33675196f38713b54694d8c2db04062e8b42a01e7655cdc039b05db
Jul 27 21:23:57.117382 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:57.117343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn" event={"ID":"556d9899-35a8-4b45-b682-2395bdedd2cc","Type":"ContainerStarted","Data":"9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b"}
Jul 27 21:23:57.117382 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:57.117386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn" event={"ID":"556d9899-35a8-4b45-b682-2395bdedd2cc","Type":"ContainerStarted","Data":"131362ced33675196f38713b54694d8c2db04062e8b42a01e7655cdc039b05db"}
Jul 27 21:23:57.141206 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:57.135472    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn" podStartSLOduration=2.135453194 podStartE2EDuration="2.135453194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:57.134743739 +0000 UTC m=+1954.500309597" watchObservedRunningTime="2026-07-27 21:23:57.135453194 +0000 UTC m=+1954.501019047"
Jul 27 21:23:58.961312 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:23:58.961283    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:24:06.142084 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:06.142038    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:24:06.142668 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:06.142099    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:24:06.154782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:06.154753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:24:06.173022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:06.172992    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:24:14.747774 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:14.747693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:24:38.349824 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:38.349791    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:24:39.755312 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.755276    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"]
Jul 27 21:24:39.755815 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.755574    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m" podUID="12fc325e-a764-46be-955b-068a23b77aab" containerName="maas-api" containerID="cri-o://6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab" gracePeriod=30
Jul 27 21:24:39.785748 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.785718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:39.813907 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.813876    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"]
Jul 27 21:24:39.814125 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.814101    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k" podUID="b9fccb0d-37ab-403a-912c-d52f68f31b71" containerName="payload-processing" containerID="cri-o://10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33" gracePeriod=30
Jul 27 21:24:39.821238 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.821207    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"]
Jul 27 21:24:39.821944 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.821415    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6" podUID="15d25fe2-d44d-4018-a420-3b9c76f868ad" containerName="payload-pre-processing" containerID="cri-o://cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551" gracePeriod=30
Jul 27 21:24:39.855840 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.855810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:39.917387 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:39.917349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:40.067317 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.067290    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:24:40.098962 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.098934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/12fc325e-a764-46be-955b-068a23b77aab-kube-api-access-qn92l\") pod \"12fc325e-a764-46be-955b-068a23b77aab\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") "
Jul 27 21:24:40.099115 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.099035    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fc325e-a764-46be-955b-068a23b77aab-openshift-service-ca\") pod \"12fc325e-a764-46be-955b-068a23b77aab\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") "
Jul 27 21:24:40.099115 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.099065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fc325e-a764-46be-955b-068a23b77aab-maas-api-tls\") pod \"12fc325e-a764-46be-955b-068a23b77aab\" (UID: \"12fc325e-a764-46be-955b-068a23b77aab\") "
Jul 27 21:24:40.099921 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.099529    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12fc325e-a764-46be-955b-068a23b77aab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12fc325e-a764-46be-955b-068a23b77aab" (UID: "12fc325e-a764-46be-955b-068a23b77aab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:40.101365 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.101339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fc325e-a764-46be-955b-068a23b77aab-kube-api-access-qn92l" (OuterVolumeSpecName: "kube-api-access-qn92l") pod "12fc325e-a764-46be-955b-068a23b77aab" (UID: "12fc325e-a764-46be-955b-068a23b77aab"). InnerVolumeSpecName "kube-api-access-qn92l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:40.101461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.101398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12fc325e-a764-46be-955b-068a23b77aab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12fc325e-a764-46be-955b-068a23b77aab" (UID: "12fc325e-a764-46be-955b-068a23b77aab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:24:40.106945 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.106924    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:24:40.109967 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.109948    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:24:40.199764 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.199728    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47zl7\" (UniqueName: \"kubernetes.io/projected/15d25fe2-d44d-4018-a420-3b9c76f868ad-kube-api-access-47zl7\") pod \"15d25fe2-d44d-4018-a420-3b9c76f868ad\" (UID: \"15d25fe2-d44d-4018-a420-3b9c76f868ad\") "
Jul 27 21:24:40.199764 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.199770    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d25fe2-d44d-4018-a420-3b9c76f868ad-plugins-config-volume\") pod \"15d25fe2-d44d-4018-a420-3b9c76f868ad\" (UID: \"15d25fe2-d44d-4018-a420-3b9c76f868ad\") "
Jul 27 21:24:40.199984 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.199834    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t57n\" (UniqueName: \"kubernetes.io/projected/b9fccb0d-37ab-403a-912c-d52f68f31b71-kube-api-access-7t57n\") pod \"b9fccb0d-37ab-403a-912c-d52f68f31b71\" (UID: \"b9fccb0d-37ab-403a-912c-d52f68f31b71\") "
Jul 27 21:24:40.199984 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.199868    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fccb0d-37ab-403a-912c-d52f68f31b71-plugins-config-volume\") pod \"b9fccb0d-37ab-403a-912c-d52f68f31b71\" (UID: \"b9fccb0d-37ab-403a-912c-d52f68f31b71\") "
Jul 27 21:24:40.200146 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.200129    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn92l\" (UniqueName: \"kubernetes.io/projected/12fc325e-a764-46be-955b-068a23b77aab-kube-api-access-qn92l\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:24:40.200219 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.200152    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fc325e-a764-46be-955b-068a23b77aab-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:24:40.200219 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.200167    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12fc325e-a764-46be-955b-068a23b77aab-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:24:40.200324 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.200253    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d25fe2-d44d-4018-a420-3b9c76f868ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15d25fe2-d44d-4018-a420-3b9c76f868ad" (UID: "15d25fe2-d44d-4018-a420-3b9c76f868ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:40.200324 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.200280    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fccb0d-37ab-403a-912c-d52f68f31b71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b9fccb0d-37ab-403a-912c-d52f68f31b71" (UID: "b9fccb0d-37ab-403a-912c-d52f68f31b71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:40.202010 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.201988    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fccb0d-37ab-403a-912c-d52f68f31b71-kube-api-access-7t57n" (OuterVolumeSpecName: "kube-api-access-7t57n") pod "b9fccb0d-37ab-403a-912c-d52f68f31b71" (UID: "b9fccb0d-37ab-403a-912c-d52f68f31b71"). InnerVolumeSpecName "kube-api-access-7t57n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:40.202063 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.201996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d25fe2-d44d-4018-a420-3b9c76f868ad-kube-api-access-47zl7" (OuterVolumeSpecName: "kube-api-access-47zl7") pod "15d25fe2-d44d-4018-a420-3b9c76f868ad" (UID: "15d25fe2-d44d-4018-a420-3b9c76f868ad"). InnerVolumeSpecName "kube-api-access-47zl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:40.301647 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.301557    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47zl7\" (UniqueName: \"kubernetes.io/projected/15d25fe2-d44d-4018-a420-3b9c76f868ad-kube-api-access-47zl7\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:24:40.301647 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.301589    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d25fe2-d44d-4018-a420-3b9c76f868ad-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:24:40.301647 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.301598    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t57n\" (UniqueName: \"kubernetes.io/projected/b9fccb0d-37ab-403a-912c-d52f68f31b71-kube-api-access-7t57n\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:24:40.301647 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.301607    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9fccb0d-37ab-403a-912c-d52f68f31b71-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:24:40.320869 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.320836    2577 generic.go:356] "Generic (PLEG): container finished" podID="15d25fe2-d44d-4018-a420-3b9c76f868ad" containerID="cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551" exitCode=0
Jul 27 21:24:40.321049 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.320883    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"
Jul 27 21:24:40.321049 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.320932    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6" event={"ID":"15d25fe2-d44d-4018-a420-3b9c76f868ad","Type":"ContainerDied","Data":"cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551"}
Jul 27 21:24:40.321049 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.320987    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6" event={"ID":"15d25fe2-d44d-4018-a420-3b9c76f868ad","Type":"ContainerDied","Data":"9f33a165bdad4c3f2e820482506392009f567813f0ec9dba0e85258ac2f0afec"}
Jul 27 21:24:40.321049 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.321017    2577 scope.go:117] "RemoveContainer" containerID="cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551"
Jul 27 21:24:40.322261 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.322241    2577 generic.go:356] "Generic (PLEG): container finished" podID="b9fccb0d-37ab-403a-912c-d52f68f31b71" containerID="10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33" exitCode=0
Jul 27 21:24:40.322364 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.322270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k" event={"ID":"b9fccb0d-37ab-403a-912c-d52f68f31b71","Type":"ContainerDied","Data":"10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33"}
Jul 27 21:24:40.322364 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.322295    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"
Jul 27 21:24:40.322364 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.322324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k" event={"ID":"b9fccb0d-37ab-403a-912c-d52f68f31b71","Type":"ContainerDied","Data":"39d8085c5872582e4fd2864592d50f38c8c814d43dc0b562c115da63e86080f0"}
Jul 27 21:24:40.323690 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.323669    2577 generic.go:356] "Generic (PLEG): container finished" podID="12fc325e-a764-46be-955b-068a23b77aab" containerID="6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab" exitCode=0
Jul 27 21:24:40.323792 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.323711    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"
Jul 27 21:24:40.323792 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.323737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m" event={"ID":"12fc325e-a764-46be-955b-068a23b77aab","Type":"ContainerDied","Data":"6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab"}
Jul 27 21:24:40.323792 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.323788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m" event={"ID":"12fc325e-a764-46be-955b-068a23b77aab","Type":"ContainerDied","Data":"b7121f750c65d501d498e1d57fed5f6f23d4e8dc780ebb151004b21416d26e49"}
Jul 27 21:24:40.330135 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.330118    2577 scope.go:117] "RemoveContainer" containerID="cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551"
Jul 27 21:24:40.330443 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:24:40.330420    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551\": container with ID starting with cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551 not found: ID does not exist" containerID="cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551"
Jul 27 21:24:40.330528 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.330450    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551"} err="failed to get container status \"cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551\": rpc error: code = NotFound desc = could not find container \"cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551\": container with ID starting with cc13142ab67d70d066ad5493b23f41a05454dffece7687b3dbf04af719afd551 not found: ID does not exist"
Jul 27 21:24:40.330528 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.330467    2577 scope.go:117] "RemoveContainer" containerID="10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33"
Jul 27 21:24:40.339613 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.339598    2577 scope.go:117] "RemoveContainer" containerID="10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33"
Jul 27 21:24:40.339857 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:24:40.339841    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33\": container with ID starting with 10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33 not found: ID does not exist" containerID="10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33"
Jul 27 21:24:40.339906 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.339862    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33"} err="failed to get container status \"10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33\": rpc error: code = NotFound desc = could not find container \"10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33\": container with ID starting with 10ccc7ddc3894c1314623f2865f2e2e38090b44accd50bad9d63763a484faf33 not found: ID does not exist"
Jul 27 21:24:40.339906 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.339876    2577 scope.go:117] "RemoveContainer" containerID="6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab"
Jul 27 21:24:40.347713 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.347690    2577 scope.go:117] "RemoveContainer" containerID="6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab"
Jul 27 21:24:40.347981 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:24:40.347965    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab\": container with ID starting with 6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab not found: ID does not exist" containerID="6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab"
Jul 27 21:24:40.348041 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.347986    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab"} err="failed to get container status \"6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab\": rpc error: code = NotFound desc = could not find container \"6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab\": container with ID starting with 6ff6ac29f1d841f9385ec2ac85e88a55be67446de2e90046594c7ba52f485dab not found: ID does not exist"
Jul 27 21:24:40.348796 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.348777    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"]
Jul 27 21:24:40.354916 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.354895    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-316c1e32-f6d5f8c95-b6p9m"]
Jul 27 21:24:40.365130 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.365096    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"]
Jul 27 21:24:40.368510 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.368492    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-316c1e32-5f7846d-mtgj6"]
Jul 27 21:24:40.378765 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.378744    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"]
Jul 27 21:24:40.382969 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:40.382948    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-316c1e32-6947444d58-hq94k"]
Jul 27 21:24:41.185067 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:41.184984    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12fc325e-a764-46be-955b-068a23b77aab" path="/var/lib/kubelet/pods/12fc325e-a764-46be-955b-068a23b77aab/volumes"
Jul 27 21:24:41.185467 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:41.185350    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15d25fe2-d44d-4018-a420-3b9c76f868ad" path="/var/lib/kubelet/pods/15d25fe2-d44d-4018-a420-3b9c76f868ad/volumes"
Jul 27 21:24:41.185657 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:41.185644    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9fccb0d-37ab-403a-912c-d52f68f31b71" path="/var/lib/kubelet/pods/b9fccb0d-37ab-403a-912c-d52f68f31b71/volumes"
Jul 27 21:24:44.372361 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:44.372328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:46.753449 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:46.753414    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:24:47.433302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:47.433263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:52.413441 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:52.413413    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"]
Jul 27 21:24:52.510931 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:52.510893    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:24:56.283179 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:24:56.283142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:07.432986 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.432939    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn" podUID="556d9899-35a8-4b45-b682-2395bdedd2cc" containerName="main" containerID="cri-o://9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b" gracePeriod=45
Jul 27 21:25:07.679645 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.679621    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:25:07.842319 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842290    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-tmp-dir\") pod \"556d9899-35a8-4b45-b682-2395bdedd2cc\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") "
Jul 27 21:25:07.842518 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842331    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-dshm\") pod \"556d9899-35a8-4b45-b682-2395bdedd2cc\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") "
Jul 27 21:25:07.842518 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842352    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-home\") pod \"556d9899-35a8-4b45-b682-2395bdedd2cc\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") "
Jul 27 21:25:07.842518 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842368    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cn9q\" (UniqueName: \"kubernetes.io/projected/556d9899-35a8-4b45-b682-2395bdedd2cc-kube-api-access-8cn9q\") pod \"556d9899-35a8-4b45-b682-2395bdedd2cc\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") "
Jul 27 21:25:07.842518 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842436    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-model-cache\") pod \"556d9899-35a8-4b45-b682-2395bdedd2cc\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") "
Jul 27 21:25:07.842518 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842459    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556d9899-35a8-4b45-b682-2395bdedd2cc-tls-certs\") pod \"556d9899-35a8-4b45-b682-2395bdedd2cc\" (UID: \"556d9899-35a8-4b45-b682-2395bdedd2cc\") "
Jul 27 21:25:07.842768 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842593    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "556d9899-35a8-4b45-b682-2395bdedd2cc" (UID: "556d9899-35a8-4b45-b682-2395bdedd2cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:07.842768 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842609    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-home" (OuterVolumeSpecName: "home") pod "556d9899-35a8-4b45-b682-2395bdedd2cc" (UID: "556d9899-35a8-4b45-b682-2395bdedd2cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:07.842768 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842710    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "556d9899-35a8-4b45-b682-2395bdedd2cc" (UID: "556d9899-35a8-4b45-b682-2395bdedd2cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:07.842871 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842766    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-tmp-dir\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:25:07.842871 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.842785    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-home\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:25:07.844804 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.844763    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556d9899-35a8-4b45-b682-2395bdedd2cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "556d9899-35a8-4b45-b682-2395bdedd2cc" (UID: "556d9899-35a8-4b45-b682-2395bdedd2cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:25:07.844907 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.844838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-dshm" (OuterVolumeSpecName: "dshm") pod "556d9899-35a8-4b45-b682-2395bdedd2cc" (UID: "556d9899-35a8-4b45-b682-2395bdedd2cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:25:07.844907 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.844842    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556d9899-35a8-4b45-b682-2395bdedd2cc-kube-api-access-8cn9q" (OuterVolumeSpecName: "kube-api-access-8cn9q") pod "556d9899-35a8-4b45-b682-2395bdedd2cc" (UID: "556d9899-35a8-4b45-b682-2395bdedd2cc"). InnerVolumeSpecName "kube-api-access-8cn9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:25:07.943209 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.943143    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-dshm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:25:07.943209 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.943176    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cn9q\" (UniqueName: \"kubernetes.io/projected/556d9899-35a8-4b45-b682-2395bdedd2cc-kube-api-access-8cn9q\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:25:07.943209 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.943216    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/556d9899-35a8-4b45-b682-2395bdedd2cc-model-cache\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:25:07.943468 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:07.943227    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/556d9899-35a8-4b45-b682-2395bdedd2cc-tls-certs\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:25:08.463309 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.463272    2577 generic.go:356] "Generic (PLEG): container finished" podID="556d9899-35a8-4b45-b682-2395bdedd2cc" containerID="9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b" exitCode=0
Jul 27 21:25:08.463717 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.463316    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"
Jul 27 21:25:08.463717 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.463324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn" event={"ID":"556d9899-35a8-4b45-b682-2395bdedd2cc","Type":"ContainerDied","Data":"9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b"}
Jul 27 21:25:08.463717 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.463354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn" event={"ID":"556d9899-35a8-4b45-b682-2395bdedd2cc","Type":"ContainerDied","Data":"131362ced33675196f38713b54694d8c2db04062e8b42a01e7655cdc039b05db"}
Jul 27 21:25:08.463717 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.463370    2577 scope.go:117] "RemoveContainer" containerID="9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b"
Jul 27 21:25:08.472540 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.472520    2577 scope.go:117] "RemoveContainer" containerID="9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b"
Jul 27 21:25:08.472786 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:25:08.472768    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b\": container with ID starting with 9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b not found: ID does not exist" containerID="9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b"
Jul 27 21:25:08.472835 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.472794    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b"} err="failed to get container status \"9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b\": rpc error: code = NotFound desc = could not find container \"9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b\": container with ID starting with 9a5588be625d11647a527565f3072fa4811c850419b9edd07ab4c04a6a6c362b not found: ID does not exist"
Jul 27 21:25:08.484164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.484139    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"]
Jul 27 21:25:08.487593 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:08.487571    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-316c1e32/e2e-lifecycle-model-316c1e32-kserve-7cf84ff467-p27vn"]
Jul 27 21:25:09.184406 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:09.184375    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="556d9899-35a8-4b45-b682-2395bdedd2cc" path="/var/lib/kubelet/pods/556d9899-35a8-4b45-b682-2395bdedd2cc/volumes"
Jul 27 21:25:12.460036 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:12.459998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:12.517214 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:12.517156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:23.311016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:23.310984    2577 scope.go:117] "RemoveContainer" containerID="e6bab33f6d8b11bd52a1d6a28329c7b89169b550aeb2b7d16890d861f20fc2fc"
Jul 27 21:25:29.002905 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:29.002869    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:29.352616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:29.352494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:34.907538 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:34.907497    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:40.375971 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:40.375932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:40.476238 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:40.476169    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:40.651500 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:40.651399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:40.880597 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:40.880544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:41.922838 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:41.922797    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:42.024591 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:42.024554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:42.124878 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:42.124842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:42.835131 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:42.835090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:42.934924 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:42.934873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:43.034852 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:43.034817    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:44.329566 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:44.329533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:44.555499 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:44.555463    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:25:50.040025 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:50.039985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:50.171279 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:50.171245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:50.690853 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:50.690811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:56.311082 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:56.311042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:56.417045 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:56.416991    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:56.512394 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:56.512346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:25:57.249519 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:25:57.249484    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:26:06.992349 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:06.992306    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:07.339567 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:07.339447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:23.343404 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:23.343371    2577 scope.go:117] "RemoveContainer" containerID="1e5b598767a55566fc615d78150185f2929438cf17447dcbcdd2b6360a05c9e5"
Jul 27 21:26:28.265590 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:28.265549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:34.603141 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:34.603107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:34.699557 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:34.699520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:34.928482 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:34.928381    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:35.237511 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:35.237480    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:35.534393 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:35.534312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:38.225572 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.225533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:38.348709 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.348679    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"]
Jul 27 21:26:38.349024 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349012    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12fc325e-a764-46be-955b-068a23b77aab" containerName="maas-api"
Jul 27 21:26:38.349024 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349025    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="12fc325e-a764-46be-955b-068a23b77aab" containerName="maas-api"
Jul 27 21:26:38.349107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349042    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="556d9899-35a8-4b45-b682-2395bdedd2cc" containerName="main"
Jul 27 21:26:38.349107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349048    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="556d9899-35a8-4b45-b682-2395bdedd2cc" containerName="main"
Jul 27 21:26:38.349107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349062    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d25fe2-d44d-4018-a420-3b9c76f868ad" containerName="payload-pre-processing"
Jul 27 21:26:38.349107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349067    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d25fe2-d44d-4018-a420-3b9c76f868ad" containerName="payload-pre-processing"
Jul 27 21:26:38.349107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349077    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9fccb0d-37ab-403a-912c-d52f68f31b71" containerName="payload-processing"
Jul 27 21:26:38.349107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349082    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9fccb0d-37ab-403a-912c-d52f68f31b71" containerName="payload-processing"
Jul 27 21:26:38.349312 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349129    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d25fe2-d44d-4018-a420-3b9c76f868ad" containerName="payload-pre-processing"
Jul 27 21:26:38.349312 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349135    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9fccb0d-37ab-403a-912c-d52f68f31b71" containerName="payload-processing"
Jul 27 21:26:38.349312 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349144    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="12fc325e-a764-46be-955b-068a23b77aab" containerName="maas-api"
Jul 27 21:26:38.349312 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.349152    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="556d9899-35a8-4b45-b682-2395bdedd2cc" containerName="main"
Jul 27 21:26:38.351904 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.351889    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.355109 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.355089    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:26:38.355490 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.355473    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:26:38.356340 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.356320    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-edfab0-serving-cert\""
Jul 27 21:26:38.356340 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.356335    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cgjq6\""
Jul 27 21:26:38.357039 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.356994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:38.368678 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.368655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"]
Jul 27 21:26:38.435614 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.435581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba416c6-6276-49f6-b52a-389b965874db-maas-api-tls\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.435796 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.435642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba416c6-6276-49f6-b52a-389b965874db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.435796 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.435709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b58g5\" (UniqueName: \"kubernetes.io/projected/8ba416c6-6276-49f6-b52a-389b965874db-kube-api-access-b58g5\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.445241 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.445204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:38.536873 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.536836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba416c6-6276-49f6-b52a-389b965874db-maas-api-tls\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.537096 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.536888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba416c6-6276-49f6-b52a-389b965874db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.537096 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.536928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b58g5\" (UniqueName: \"kubernetes.io/projected/8ba416c6-6276-49f6-b52a-389b965874db-kube-api-access-b58g5\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.537680 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.537621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba416c6-6276-49f6-b52a-389b965874db-openshift-service-ca\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.539524 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.539506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba416c6-6276-49f6-b52a-389b965874db-maas-api-tls\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.546078 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.546059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b58g5\" (UniqueName: \"kubernetes.io/projected/8ba416c6-6276-49f6-b52a-389b965874db-kube-api-access-b58g5\") pod \"maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.566267 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.566235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:38.662114 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.662078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:38.816092 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.816011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"]
Jul 27 21:26:38.819210 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:26:38.819170    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba416c6_6276_49f6_b52a_389b965874db.slice/crio-b8b7a9f1b0da10cf066c1286ba2b0d76dc98fc915635728886fdc59f810319c7 WatchSource:0}: Error finding container b8b7a9f1b0da10cf066c1286ba2b0d76dc98fc915635728886fdc59f810319c7: Status 404 returned error can't find the container with id b8b7a9f1b0da10cf066c1286ba2b0d76dc98fc915635728886fdc59f810319c7
Jul 27 21:26:38.820610 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.820592    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:26:38.867811 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:38.867784    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd" event={"ID":"8ba416c6-6276-49f6-b52a-389b965874db","Type":"ContainerStarted","Data":"b8b7a9f1b0da10cf066c1286ba2b0d76dc98fc915635728886fdc59f810319c7"}
Jul 27 21:26:39.048919 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.048890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:39.153446 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.153356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:39.163201 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.163164    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"]
Jul 27 21:26:39.167289 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.167271    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"]
Jul 27 21:26:39.167427 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.167408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.171155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.171134    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-edfab0-dockercfg-r2qlc\""
Jul 27 21:26:39.171300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.171284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.171361 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.171335    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-edfab0\""
Jul 27 21:26:39.185725 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.185708    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"]
Jul 27 21:26:39.185837 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.185751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"]
Jul 27 21:26:39.242460 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.242426    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r6hx\" (UniqueName: \"kubernetes.io/projected/6bac165b-ced5-4020-9599-67006301128a-kube-api-access-4r6hx\") pod \"payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr\" (UID: \"6bac165b-ced5-4020-9599-67006301128a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.242846 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.242471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac165b-ced5-4020-9599-67006301128a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr\" (UID: \"6bac165b-ced5-4020-9599-67006301128a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.242846 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.242489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9zlq\" (UniqueName: \"kubernetes.io/projected/cd65d57f-9497-45f9-bf79-3c4ae16565bf-kube-api-access-m9zlq\") pod \"payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn\" (UID: \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.242846 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.242566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd65d57f-9497-45f9-bf79-3c4ae16565bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn\" (UID: \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.343014 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.342981    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac165b-ced5-4020-9599-67006301128a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr\" (UID: \"6bac165b-ced5-4020-9599-67006301128a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.343014 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.343012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9zlq\" (UniqueName: \"kubernetes.io/projected/cd65d57f-9497-45f9-bf79-3c4ae16565bf-kube-api-access-m9zlq\") pod \"payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn\" (UID: \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.343247 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.343036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd65d57f-9497-45f9-bf79-3c4ae16565bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn\" (UID: \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.343247 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.343113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r6hx\" (UniqueName: \"kubernetes.io/projected/6bac165b-ced5-4020-9599-67006301128a-kube-api-access-4r6hx\") pod \"payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr\" (UID: \"6bac165b-ced5-4020-9599-67006301128a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.343933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.343911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac165b-ced5-4020-9599-67006301128a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr\" (UID: \"6bac165b-ced5-4020-9599-67006301128a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.343970 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.343911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd65d57f-9497-45f9-bf79-3c4ae16565bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn\" (UID: \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.358280 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.358235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9zlq\" (UniqueName: \"kubernetes.io/projected/cd65d57f-9497-45f9-bf79-3c4ae16565bf-kube-api-access-m9zlq\") pod \"payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn\" (UID: \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.359855 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.359836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r6hx\" (UniqueName: \"kubernetes.io/projected/6bac165b-ced5-4020-9599-67006301128a-kube-api-access-4r6hx\") pod \"payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr\" (UID: \"6bac165b-ced5-4020-9599-67006301128a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.479263 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.479240    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.488779 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.488754    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.625725 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.625703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"]
Jul 27 21:26:39.627799 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:26:39.627773    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd65d57f_9497_45f9_bf79_3c4ae16565bf.slice/crio-a8018591fa99257f5d4897c16597d95519378608a39ec5bb8ad12e836117db04 WatchSource:0}: Error finding container a8018591fa99257f5d4897c16597d95519378608a39ec5bb8ad12e836117db04: Status 404 returned error can't find the container with id a8018591fa99257f5d4897c16597d95519378608a39ec5bb8ad12e836117db04
Jul 27 21:26:39.653550 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.653530    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"]
Jul 27 21:26:39.661358 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:26:39.661335    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bac165b_ced5_4020_9599_67006301128a.slice/crio-5796f83f19014de34f99a9576275124ed11819d13b1fc714d0c268985d047f3d WatchSource:0}: Error finding container 5796f83f19014de34f99a9576275124ed11819d13b1fc714d0c268985d047f3d: Status 404 returned error can't find the container with id 5796f83f19014de34f99a9576275124ed11819d13b1fc714d0c268985d047f3d
Jul 27 21:26:39.876476 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.876446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr" event={"ID":"6bac165b-ced5-4020-9599-67006301128a","Type":"ContainerStarted","Data":"8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd"}
Jul 27 21:26:39.876656 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.876497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr" event={"ID":"6bac165b-ced5-4020-9599-67006301128a","Type":"ContainerStarted","Data":"5796f83f19014de34f99a9576275124ed11819d13b1fc714d0c268985d047f3d"}
Jul 27 21:26:39.876656 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.876540    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:39.878046 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.878020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd" event={"ID":"8ba416c6-6276-49f6-b52a-389b965874db","Type":"ContainerStarted","Data":"8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7"}
Jul 27 21:26:39.878244 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.878129    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:39.879582 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.879563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn" event={"ID":"cd65d57f-9497-45f9-bf79-3c4ae16565bf","Type":"ContainerStarted","Data":"72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6"}
Jul 27 21:26:39.879687 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.879596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn" event={"ID":"cd65d57f-9497-45f9-bf79-3c4ae16565bf","Type":"ContainerStarted","Data":"a8018591fa99257f5d4897c16597d95519378608a39ec5bb8ad12e836117db04"}
Jul 27 21:26:39.879742 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.879689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:39.911802 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.911754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr" podStartSLOduration=0.911740769 podStartE2EDuration="911.740769ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:39.910230874 +0000 UTC m=+2117.275796723" watchObservedRunningTime="2026-07-27 21:26:39.911740769 +0000 UTC m=+2117.277306594"
Jul 27 21:26:39.934329 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.934251    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn" podStartSLOduration=0.934237843 podStartE2EDuration="934.237843ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:39.932650925 +0000 UTC m=+2117.298216769" watchObservedRunningTime="2026-07-27 21:26:39.934237843 +0000 UTC m=+2117.299803735"
Jul 27 21:26:39.960076 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:39.960035    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd" podStartSLOduration=1.352451491 podStartE2EDuration="1.960021278s" totalImagesPullingTime="607.569787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:39.956826287 +0000 UTC m=+2117.322392135" watchObservedRunningTime="2026-07-27 21:26:39.960021278 +0000 UTC m=+2117.325587125"
Jul 27 21:26:42.080862 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:42.080814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:26:42.730734 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:42.730232    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:43.051048 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:43.050945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:43.386007 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:43.385915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:43.518097 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:43.518046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:45.891450 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:45.891419    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:26:46.477170 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:46.477132    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:46.635564 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:46.635522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:46.863543 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:46.863512    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"]
Jul 27 21:26:46.867161 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:46.867145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:46.872306 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:46.872283    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0adc64-serving-cert\""
Jul 27 21:26:46.887844 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:46.887814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"]
Jul 27 21:26:46.906311 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:46.906277    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:47.017532 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.017502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8995\" (UniqueName: \"kubernetes.io/projected/2e1daa4e-480b-484c-ad81-718496d68db0-kube-api-access-b8995\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.017692 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.017537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1daa4e-480b-484c-ad81-718496d68db0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.017692 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.017610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1daa4e-480b-484c-ad81-718496d68db0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.056895 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.056865    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:47.119149 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.119071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1daa4e-480b-484c-ad81-718496d68db0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.119287 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.119151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8995\" (UniqueName: \"kubernetes.io/projected/2e1daa4e-480b-484c-ad81-718496d68db0-kube-api-access-b8995\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.119287 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.119210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1daa4e-480b-484c-ad81-718496d68db0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.119836 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.119815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1daa4e-480b-484c-ad81-718496d68db0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.121639 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.121614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1daa4e-480b-484c-ad81-718496d68db0-maas-api-tls\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.150005 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.149983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8995\" (UniqueName: \"kubernetes.io/projected/2e1daa4e-480b-484c-ad81-718496d68db0-kube-api-access-b8995\") pod \"maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.177859 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.177830    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.318119 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.318096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"]
Jul 27 21:26:47.320493 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:26:47.320465    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e1daa4e_480b_484c_ad81_718496d68db0.slice/crio-8f319854c365104be85a122ad28bbb387af550cc5cae730821818ec39679251b WatchSource:0}: Error finding container 8f319854c365104be85a122ad28bbb387af550cc5cae730821818ec39679251b: Status 404 returned error can't find the container with id 8f319854c365104be85a122ad28bbb387af550cc5cae730821818ec39679251b
Jul 27 21:26:47.760527 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.760498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:47.900707 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.900672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:47.921765 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.921730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8" event={"ID":"2e1daa4e-480b-484c-ad81-718496d68db0","Type":"ContainerStarted","Data":"88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d"}
Jul 27 21:26:47.922232 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.921794    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:47.922232 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.921817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8" event={"ID":"2e1daa4e-480b-484c-ad81-718496d68db0","Type":"ContainerStarted","Data":"8f319854c365104be85a122ad28bbb387af550cc5cae730821818ec39679251b"}
Jul 27 21:26:47.964821 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:47.964772    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8" podStartSLOduration=1.549618251 podStartE2EDuration="1.964754871s" totalImagesPullingTime="415.13662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:47.963789102 +0000 UTC m=+2125.329354948" watchObservedRunningTime="2026-07-27 21:26:47.964754871 +0000 UTC m=+2125.330320696"
Jul 27 21:26:48.059484 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.059411    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"]
Jul 27 21:26:48.062809 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.062783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.067760 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.067741    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0adc64\""
Jul 27 21:26:48.067840 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.067785    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0adc64-dockercfg-wdprc\""
Jul 27 21:26:48.116171 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.116140    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"]
Jul 27 21:26:48.229159 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.229127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09cbfbe-1811-445c-85a7-0e4a257d92d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz\" (UID: \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.229368 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.229170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4zg\" (UniqueName: \"kubernetes.io/projected/e09cbfbe-1811-445c-85a7-0e4a257d92d8-kube-api-access-db4zg\") pod \"payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz\" (UID: \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.330541 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.330374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09cbfbe-1811-445c-85a7-0e4a257d92d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz\" (UID: \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.330541 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.330419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4zg\" (UniqueName: \"kubernetes.io/projected/e09cbfbe-1811-445c-85a7-0e4a257d92d8-kube-api-access-db4zg\") pod \"payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz\" (UID: \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.350786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.331112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09cbfbe-1811-445c-85a7-0e4a257d92d8-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz\" (UID: \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.350786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.339862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4zg\" (UniqueName: \"kubernetes.io/projected/e09cbfbe-1811-445c-85a7-0e4a257d92d8-kube-api-access-db4zg\") pod \"payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz\" (UID: \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.372900 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.372865    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.514002 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.513980    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"]
Jul 27 21:26:48.515813 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:26:48.515786    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09cbfbe_1811_445c_85a7_0e4a257d92d8.slice/crio-e5758f8c32ed9968b40eebe8455b2075c4c8d9213d589f88ae0ae07b78ab36a0 WatchSource:0}: Error finding container e5758f8c32ed9968b40eebe8455b2075c4c8d9213d589f88ae0ae07b78ab36a0: Status 404 returned error can't find the container with id e5758f8c32ed9968b40eebe8455b2075c4c8d9213d589f88ae0ae07b78ab36a0
Jul 27 21:26:48.928411 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.928377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz" event={"ID":"e09cbfbe-1811-445c-85a7-0e4a257d92d8","Type":"ContainerStarted","Data":"411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e"}
Jul 27 21:26:48.928861 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.928475    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:48.928861 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.928515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz" event={"ID":"e09cbfbe-1811-445c-85a7-0e4a257d92d8","Type":"ContainerStarted","Data":"e5758f8c32ed9968b40eebe8455b2075c4c8d9213d589f88ae0ae07b78ab36a0"}
Jul 27 21:26:48.948521 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:48.948473    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz" podStartSLOduration=0.948457146 podStartE2EDuration="948.457146ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:26:48.947755546 +0000 UTC m=+2126.313321393" watchObservedRunningTime="2026-07-27 21:26:48.948457146 +0000 UTC m=+2126.314022992"
Jul 27 21:26:50.859799 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:50.859756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:50.888248 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:50.888220    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:26:50.888436 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:50.888348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:26:53.663241 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:53.663207    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:26:53.934053 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:53.933984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:26:59.421947 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.421907    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"]
Jul 27 21:26:59.426008 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.425984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.429915 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.429892    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-edfab0\"/\"kube-root-ca.crt\""
Jul 27 21:26:59.430082 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.430059    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-edfab0\"/\"default-dockercfg-glzqp\""
Jul 27 21:26:59.430239 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.430149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-edfab0\"/\"test-model-edfab0-kserve-self-signed-certs\""
Jul 27 21:26:59.430310 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.429984    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-edfab0\"/\"openshift-service-ca.crt\""
Jul 27 21:26:59.435048 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.435026    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"]
Jul 27 21:26:59.460403 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.460361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:26:59.526103 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.526062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqcwz\" (UniqueName: \"kubernetes.io/projected/f2e4de99-f689-4a06-b411-25f800c30401-kube-api-access-fqcwz\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.526103 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.526108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e4de99-f689-4a06-b411-25f800c30401-tls-certs\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.526378 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.526209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-dshm\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.526378 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.526228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-model-cache\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.526378 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.526253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-home\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.526378 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.526269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-tmp-dir\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.626680 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.626640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-dshm\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.626680 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.626678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-model-cache\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.626933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.626700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-home\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.626933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.626816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-tmp-dir\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.626933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.626875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqcwz\" (UniqueName: \"kubernetes.io/projected/f2e4de99-f689-4a06-b411-25f800c30401-kube-api-access-fqcwz\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.626933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.626914    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e4de99-f689-4a06-b411-25f800c30401-tls-certs\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.627133 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.627111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-model-cache\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.627133 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.627120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-home\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.627295 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.627275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-tmp-dir\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.629166 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.629146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-dshm\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.629465 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.629450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e4de99-f689-4a06-b411-25f800c30401-tls-certs\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.636827 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.636796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqcwz\" (UniqueName: \"kubernetes.io/projected/f2e4de99-f689-4a06-b411-25f800c30401-kube-api-access-fqcwz\") pod \"test-model-edfab0-kserve-954b78cb5-sz78t\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") " pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.740035 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.740002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:26:59.876430 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.876407    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"]
Jul 27 21:26:59.878372 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:26:59.878339    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2e4de99_f689_4a06_b411_25f800c30401.slice/crio-4ec65c421f31d9159ab4c961ee1c668a7a5a839d7a694b28085d9e6f9b12abf4 WatchSource:0}: Error finding container 4ec65c421f31d9159ab4c961ee1c668a7a5a839d7a694b28085d9e6f9b12abf4: Status 404 returned error can't find the container with id 4ec65c421f31d9159ab4c961ee1c668a7a5a839d7a694b28085d9e6f9b12abf4
Jul 27 21:26:59.935853 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.935829    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:26:59.983237 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.983170    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t" event={"ID":"f2e4de99-f689-4a06-b411-25f800c30401","Type":"ContainerStarted","Data":"b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362"}
Jul 27 21:26:59.983440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:26:59.983247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t" event={"ID":"f2e4de99-f689-4a06-b411-25f800c30401","Type":"ContainerStarted","Data":"4ec65c421f31d9159ab4c961ee1c668a7a5a839d7a694b28085d9e6f9b12abf4"}
Jul 27 21:27:00.002744 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:00.002638    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t" podStartSLOduration=1.002621763 podStartE2EDuration="1.002621763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:00.000291356 +0000 UTC m=+2137.365857224" watchObservedRunningTime="2026-07-27 21:27:00.002621763 +0000 UTC m=+2137.368187611"
Jul 27 21:27:02.975989 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:02.975953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:27:09.740509 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:09.740472    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:27:09.740937 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:09.740743    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:27:09.753068 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:09.753043    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:27:10.041135 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:10.041054    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:27:13.171633 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:13.171546    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:27:33.723529 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:33.723495    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:34.080800 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:34.080766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:36.515945 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:36.515908    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:36.639642 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:36.639609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:27:37.704870 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:37.704837    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:27:52.355968 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:27:52.355932    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:28:13.685341 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:13.685302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:20.851599 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:20.851564    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"]
Jul 27 21:28:20.907159 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:20.907113    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:21.907583 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:21.907541    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:26.106315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:26.106284    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:28:32.525733 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.525700    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"]
Jul 27 21:28:32.526128 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.525892    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd" podUID="8ba416c6-6276-49f6-b52a-389b965874db" containerName="maas-api" containerID="cri-o://8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7" gracePeriod=30
Jul 27 21:28:32.575448 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.575416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:32.594821 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.594792    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"]
Jul 27 21:28:32.595054 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.595029    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn" podUID="cd65d57f-9497-45f9-bf79-3c4ae16565bf" containerName="payload-processing" containerID="cri-o://72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6" gracePeriod=30
Jul 27 21:28:32.609664 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.609253    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"]
Jul 27 21:28:32.609664 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.609489    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr" podUID="6bac165b-ced5-4020-9599-67006301128a" containerName="payload-pre-processing" containerID="cri-o://8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd" gracePeriod=30
Jul 27 21:28:32.679873 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.679839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:32.817037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.817001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:32.886579 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.886555    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:28:32.889690 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.889669    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:28:32.895827 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.895808    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:28:32.966126 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac165b-ced5-4020-9599-67006301128a-plugins-config-volume\") pod \"6bac165b-ced5-4020-9599-67006301128a\" (UID: \"6bac165b-ced5-4020-9599-67006301128a\") "
Jul 27 21:28:32.966332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966151    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd65d57f-9497-45f9-bf79-3c4ae16565bf-plugins-config-volume\") pod \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\" (UID: \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\") "
Jul 27 21:28:32.966332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba416c6-6276-49f6-b52a-389b965874db-openshift-service-ca\") pod \"8ba416c6-6276-49f6-b52a-389b965874db\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") "
Jul 27 21:28:32.966332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b58g5\" (UniqueName: \"kubernetes.io/projected/8ba416c6-6276-49f6-b52a-389b965874db-kube-api-access-b58g5\") pod \"8ba416c6-6276-49f6-b52a-389b965874db\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") "
Jul 27 21:28:32.966332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9zlq\" (UniqueName: \"kubernetes.io/projected/cd65d57f-9497-45f9-bf79-3c4ae16565bf-kube-api-access-m9zlq\") pod \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\" (UID: \"cd65d57f-9497-45f9-bf79-3c4ae16565bf\") "
Jul 27 21:28:32.966332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966321    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba416c6-6276-49f6-b52a-389b965874db-maas-api-tls\") pod \"8ba416c6-6276-49f6-b52a-389b965874db\" (UID: \"8ba416c6-6276-49f6-b52a-389b965874db\") "
Jul 27 21:28:32.966616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r6hx\" (UniqueName: \"kubernetes.io/projected/6bac165b-ced5-4020-9599-67006301128a-kube-api-access-4r6hx\") pod \"6bac165b-ced5-4020-9599-67006301128a\" (UID: \"6bac165b-ced5-4020-9599-67006301128a\") "
Jul 27 21:28:32.966616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966551    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bac165b-ced5-4020-9599-67006301128a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bac165b-ced5-4020-9599-67006301128a" (UID: "6bac165b-ced5-4020-9599-67006301128a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:28:32.966616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966592    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd65d57f-9497-45f9-bf79-3c4ae16565bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cd65d57f-9497-45f9-bf79-3c4ae16565bf" (UID: "cd65d57f-9497-45f9-bf79-3c4ae16565bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:28:32.966616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba416c6-6276-49f6-b52a-389b965874db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ba416c6-6276-49f6-b52a-389b965874db" (UID: "8ba416c6-6276-49f6-b52a-389b965874db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:28:32.966813 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.966640    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bac165b-ced5-4020-9599-67006301128a-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:32.968649 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.968615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba416c6-6276-49f6-b52a-389b965874db-kube-api-access-b58g5" (OuterVolumeSpecName: "kube-api-access-b58g5") pod "8ba416c6-6276-49f6-b52a-389b965874db" (UID: "8ba416c6-6276-49f6-b52a-389b965874db"). InnerVolumeSpecName "kube-api-access-b58g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:28:32.968649 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.968636    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd65d57f-9497-45f9-bf79-3c4ae16565bf-kube-api-access-m9zlq" (OuterVolumeSpecName: "kube-api-access-m9zlq") pod "cd65d57f-9497-45f9-bf79-3c4ae16565bf" (UID: "cd65d57f-9497-45f9-bf79-3c4ae16565bf"). InnerVolumeSpecName "kube-api-access-m9zlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:28:32.968649 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.968638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac165b-ced5-4020-9599-67006301128a-kube-api-access-4r6hx" (OuterVolumeSpecName: "kube-api-access-4r6hx") pod "6bac165b-ced5-4020-9599-67006301128a" (UID: "6bac165b-ced5-4020-9599-67006301128a"). InnerVolumeSpecName "kube-api-access-4r6hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:28:32.968852 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:32.968658    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba416c6-6276-49f6-b52a-389b965874db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ba416c6-6276-49f6-b52a-389b965874db" (UID: "8ba416c6-6276-49f6-b52a-389b965874db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:28:33.067237 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.067131    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd65d57f-9497-45f9-bf79-3c4ae16565bf-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:33.067237 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.067157    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba416c6-6276-49f6-b52a-389b965874db-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:33.067237 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.067167    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b58g5\" (UniqueName: \"kubernetes.io/projected/8ba416c6-6276-49f6-b52a-389b965874db-kube-api-access-b58g5\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:33.067237 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.067176    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9zlq\" (UniqueName: \"kubernetes.io/projected/cd65d57f-9497-45f9-bf79-3c4ae16565bf-kube-api-access-m9zlq\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:33.067237 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.067210    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba416c6-6276-49f6-b52a-389b965874db-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:33.067237 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.067218    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r6hx\" (UniqueName: \"kubernetes.io/projected/6bac165b-ced5-4020-9599-67006301128a-kube-api-access-4r6hx\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:33.439560 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.439462    2577 generic.go:356] "Generic (PLEG): container finished" podID="8ba416c6-6276-49f6-b52a-389b965874db" containerID="8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7" exitCode=0
Jul 27 21:28:33.439560 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.439524    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"
Jul 27 21:28:33.439560 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.439551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd" event={"ID":"8ba416c6-6276-49f6-b52a-389b965874db","Type":"ContainerDied","Data":"8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7"}
Jul 27 21:28:33.439829 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.439611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd" event={"ID":"8ba416c6-6276-49f6-b52a-389b965874db","Type":"ContainerDied","Data":"b8b7a9f1b0da10cf066c1286ba2b0d76dc98fc915635728886fdc59f810319c7"}
Jul 27 21:28:33.439829 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.439639    2577 scope.go:117] "RemoveContainer" containerID="8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7"
Jul 27 21:28:33.440941 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.440921    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd65d57f-9497-45f9-bf79-3c4ae16565bf" containerID="72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6" exitCode=0
Jul 27 21:28:33.441057 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.440993    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"
Jul 27 21:28:33.441057 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.441016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn" event={"ID":"cd65d57f-9497-45f9-bf79-3c4ae16565bf","Type":"ContainerDied","Data":"72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6"}
Jul 27 21:28:33.441173 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.441086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn" event={"ID":"cd65d57f-9497-45f9-bf79-3c4ae16565bf","Type":"ContainerDied","Data":"a8018591fa99257f5d4897c16597d95519378608a39ec5bb8ad12e836117db04"}
Jul 27 21:28:33.442544 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.442521    2577 generic.go:356] "Generic (PLEG): container finished" podID="6bac165b-ced5-4020-9599-67006301128a" containerID="8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd" exitCode=0
Jul 27 21:28:33.442663 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.442587    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"
Jul 27 21:28:33.442663 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.442613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr" event={"ID":"6bac165b-ced5-4020-9599-67006301128a","Type":"ContainerDied","Data":"8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd"}
Jul 27 21:28:33.442663 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.442651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr" event={"ID":"6bac165b-ced5-4020-9599-67006301128a","Type":"ContainerDied","Data":"5796f83f19014de34f99a9576275124ed11819d13b1fc714d0c268985d047f3d"}
Jul 27 21:28:33.449510 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.449494    2577 scope.go:117] "RemoveContainer" containerID="8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7"
Jul 27 21:28:33.449826 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:28:33.449801    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7\": container with ID starting with 8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7 not found: ID does not exist" containerID="8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7"
Jul 27 21:28:33.449890 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.449836    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7"} err="failed to get container status \"8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7\": rpc error: code = NotFound desc = could not find container \"8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7\": container with ID starting with 8d5418dc47236b6794977da7244044bdc73d04a0848d44acbe5d8285062bbca7 not found: ID does not exist"
Jul 27 21:28:33.449890 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.449852    2577 scope.go:117] "RemoveContainer" containerID="72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6"
Jul 27 21:28:33.457819 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.457804    2577 scope.go:117] "RemoveContainer" containerID="72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6"
Jul 27 21:28:33.458062 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:28:33.458037    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6\": container with ID starting with 72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6 not found: ID does not exist" containerID="72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6"
Jul 27 21:28:33.458153 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.458066    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6"} err="failed to get container status \"72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6\": rpc error: code = NotFound desc = could not find container \"72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6\": container with ID starting with 72d81263c0f956545afa3e530319140cd74bf112f314ba9c1c60eaaf29b845f6 not found: ID does not exist"
Jul 27 21:28:33.458153 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.458082    2577 scope.go:117] "RemoveContainer" containerID="8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd"
Jul 27 21:28:33.465253 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.465230    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"]
Jul 27 21:28:33.465681 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.465668    2577 scope.go:117] "RemoveContainer" containerID="8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd"
Jul 27 21:28:33.465936 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:28:33.465918    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd\": container with ID starting with 8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd not found: ID does not exist" containerID="8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd"
Jul 27 21:28:33.465979 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.465944    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd"} err="failed to get container status \"8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd\": rpc error: code = NotFound desc = could not find container \"8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd\": container with ID starting with 8caf0982c95ab4c6ac4e5b9bb308f00ff7a99941344a9cb6b1554302c75546cd not found: ID does not exist"
Jul 27 21:28:33.470523 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.470503    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-edfab0-754bd9794-j65fn"]
Jul 27 21:28:33.481678 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.481658    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"]
Jul 27 21:28:33.486153 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.486135    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-edfab0-7ccc67f5b4-jzpmr"]
Jul 27 21:28:33.500442 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.500417    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"]
Jul 27 21:28:33.504234 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:33.504217    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-edfab0-7b8c4b56cf-4k8nd"]
Jul 27 21:28:35.185037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:35.185001    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bac165b-ced5-4020-9599-67006301128a" path="/var/lib/kubelet/pods/6bac165b-ced5-4020-9599-67006301128a/volumes"
Jul 27 21:28:35.185409 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:35.185372    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba416c6-6276-49f6-b52a-389b965874db" path="/var/lib/kubelet/pods/8ba416c6-6276-49f6-b52a-389b965874db/volumes"
Jul 27 21:28:35.185675 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:35.185661    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd65d57f-9497-45f9-bf79-3c4ae16565bf" path="/var/lib/kubelet/pods/cd65d57f-9497-45f9-bf79-3c4ae16565bf/volumes"
Jul 27 21:28:35.870911 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:35.870875    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t" podUID="f2e4de99-f689-4a06-b411-25f800c30401" containerName="main" containerID="cri-o://b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362" gracePeriod=45
Jul 27 21:28:36.122147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.122091    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:28:36.192575 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192550    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-tmp-dir\") pod \"f2e4de99-f689-4a06-b411-25f800c30401\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") "
Jul 27 21:28:36.193022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e4de99-f689-4a06-b411-25f800c30401-tls-certs\") pod \"f2e4de99-f689-4a06-b411-25f800c30401\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") "
Jul 27 21:28:36.193022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192647    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-dshm\") pod \"f2e4de99-f689-4a06-b411-25f800c30401\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") "
Jul 27 21:28:36.193022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192666    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-model-cache\") pod \"f2e4de99-f689-4a06-b411-25f800c30401\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") "
Jul 27 21:28:36.193022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-home\") pod \"f2e4de99-f689-4a06-b411-25f800c30401\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") "
Jul 27 21:28:36.193022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192736    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqcwz\" (UniqueName: \"kubernetes.io/projected/f2e4de99-f689-4a06-b411-25f800c30401-kube-api-access-fqcwz\") pod \"f2e4de99-f689-4a06-b411-25f800c30401\" (UID: \"f2e4de99-f689-4a06-b411-25f800c30401\") "
Jul 27 21:28:36.193022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192853    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2e4de99-f689-4a06-b411-25f800c30401" (UID: "f2e4de99-f689-4a06-b411-25f800c30401"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:28:36.193022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192946    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2e4de99-f689-4a06-b411-25f800c30401" (UID: "f2e4de99-f689-4a06-b411-25f800c30401"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:28:36.193022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.192967    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-home" (OuterVolumeSpecName: "home") pod "f2e4de99-f689-4a06-b411-25f800c30401" (UID: "f2e4de99-f689-4a06-b411-25f800c30401"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:28:36.194038 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.193458    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-tmp-dir\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:36.194038 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.193478    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-model-cache\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:36.194038 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.193493    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-home\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:36.195265 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.195241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-dshm" (OuterVolumeSpecName: "dshm") pod "f2e4de99-f689-4a06-b411-25f800c30401" (UID: "f2e4de99-f689-4a06-b411-25f800c30401"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:28:36.195462 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.195442    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e4de99-f689-4a06-b411-25f800c30401-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2e4de99-f689-4a06-b411-25f800c30401" (UID: "f2e4de99-f689-4a06-b411-25f800c30401"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:28:36.195530 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.195462    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e4de99-f689-4a06-b411-25f800c30401-kube-api-access-fqcwz" (OuterVolumeSpecName: "kube-api-access-fqcwz") pod "f2e4de99-f689-4a06-b411-25f800c30401" (UID: "f2e4de99-f689-4a06-b411-25f800c30401"). InnerVolumeSpecName "kube-api-access-fqcwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:28:36.293933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.293898    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqcwz\" (UniqueName: \"kubernetes.io/projected/f2e4de99-f689-4a06-b411-25f800c30401-kube-api-access-fqcwz\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:36.293933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.293928    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2e4de99-f689-4a06-b411-25f800c30401-tls-certs\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:36.293933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.293938    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2e4de99-f689-4a06-b411-25f800c30401-dshm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:36.462611 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.462520    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2e4de99-f689-4a06-b411-25f800c30401" containerID="b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362" exitCode=0
Jul 27 21:28:36.462611 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.462572    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"
Jul 27 21:28:36.462611 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.462602    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t" event={"ID":"f2e4de99-f689-4a06-b411-25f800c30401","Type":"ContainerDied","Data":"b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362"}
Jul 27 21:28:36.462851 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.462642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t" event={"ID":"f2e4de99-f689-4a06-b411-25f800c30401","Type":"ContainerDied","Data":"4ec65c421f31d9159ab4c961ee1c668a7a5a839d7a694b28085d9e6f9b12abf4"}
Jul 27 21:28:36.462851 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.462659    2577 scope.go:117] "RemoveContainer" containerID="b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362"
Jul 27 21:28:36.472125 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.472106    2577 scope.go:117] "RemoveContainer" containerID="b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362"
Jul 27 21:28:36.472414 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:28:36.472396    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362\": container with ID starting with b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362 not found: ID does not exist" containerID="b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362"
Jul 27 21:28:36.472477 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.472424    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362"} err="failed to get container status \"b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362\": rpc error: code = NotFound desc = could not find container \"b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362\": container with ID starting with b00597063defaf1d9636b4519ceac9780ba895888b04d83fb57a2036c5925362 not found: ID does not exist"
Jul 27 21:28:36.489129 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.489105    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"]
Jul 27 21:28:36.496535 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:36.496508    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-edfab0/test-model-edfab0-kserve-954b78cb5-sz78t"]
Jul 27 21:28:37.136740 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:37.136708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:37.184984 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:37.184945    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e4de99-f689-4a06-b411-25f800c30401" path="/var/lib/kubelet/pods/f2e4de99-f689-4a06-b411-25f800c30401/volumes"
Jul 27 21:28:37.600201 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:37.600150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:41.284788 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:41.284746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:28:44.467014 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:44.466911    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:45.133740 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.133706    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"]
Jul 27 21:28:45.134128 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134115    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e4de99-f689-4a06-b411-25f800c30401" containerName="main"
Jul 27 21:28:45.134172 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134130    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e4de99-f689-4a06-b411-25f800c30401" containerName="main"
Jul 27 21:28:45.134172 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134151    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba416c6-6276-49f6-b52a-389b965874db" containerName="maas-api"
Jul 27 21:28:45.134172 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134159    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba416c6-6276-49f6-b52a-389b965874db" containerName="maas-api"
Jul 27 21:28:45.134302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134178    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bac165b-ced5-4020-9599-67006301128a" containerName="payload-pre-processing"
Jul 27 21:28:45.134302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134197    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bac165b-ced5-4020-9599-67006301128a" containerName="payload-pre-processing"
Jul 27 21:28:45.134302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134203    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd65d57f-9497-45f9-bf79-3c4ae16565bf" containerName="payload-processing"
Jul 27 21:28:45.134302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134211    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd65d57f-9497-45f9-bf79-3c4ae16565bf" containerName="payload-processing"
Jul 27 21:28:45.134302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134268    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba416c6-6276-49f6-b52a-389b965874db" containerName="maas-api"
Jul 27 21:28:45.134302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134279    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e4de99-f689-4a06-b411-25f800c30401" containerName="main"
Jul 27 21:28:45.134302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134286    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd65d57f-9497-45f9-bf79-3c4ae16565bf" containerName="payload-processing"
Jul 27 21:28:45.134302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.134292    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bac165b-ced5-4020-9599-67006301128a" containerName="payload-pre-processing"
Jul 27 21:28:45.138454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.138437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"
Jul 27 21:28:45.142278 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.142256    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fvvt\""
Jul 27 21:28:45.149426 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.149404    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"]
Jul 27 21:28:45.169803 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.169774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66024ba2-367a-4408-9381-d62f2641ae76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm\" (UID: \"66024ba2-367a-4408-9381-d62f2641ae76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"
Jul 27 21:28:45.270474 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.270431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66024ba2-367a-4408-9381-d62f2641ae76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm\" (UID: \"66024ba2-367a-4408-9381-d62f2641ae76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"
Jul 27 21:28:45.271091 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.271069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66024ba2-367a-4408-9381-d62f2641ae76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm\" (UID: \"66024ba2-367a-4408-9381-d62f2641ae76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"
Jul 27 21:28:45.449255 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.449161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"
Jul 27 21:28:45.591988 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:45.591960    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"]
Jul 27 21:28:45.594660 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:28:45.594631    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66024ba2_367a_4408_9381_d62f2641ae76.slice/crio-5ae0ddea4ff94bcd331a5c89ed579c6564b8be6c71cef3d82fcc7ba1e2ea22c3 WatchSource:0}: Error finding container 5ae0ddea4ff94bcd331a5c89ed579c6564b8be6c71cef3d82fcc7ba1e2ea22c3: Status 404 returned error can't find the container with id 5ae0ddea4ff94bcd331a5c89ed579c6564b8be6c71cef3d82fcc7ba1e2ea22c3
Jul 27 21:28:46.022013 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:46.021914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:46.515471 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:46.515436    2577 generic.go:356] "Generic (PLEG): container finished" podID="66024ba2-367a-4408-9381-d62f2641ae76" containerID="86a6c3ce76c3eee7da324523f761d5ca79aba98d8b1a7875f100ae1418c132d4" exitCode=0
Jul 27 21:28:46.515676 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:46.515487    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm" event={"ID":"66024ba2-367a-4408-9381-d62f2641ae76","Type":"ContainerDied","Data":"86a6c3ce76c3eee7da324523f761d5ca79aba98d8b1a7875f100ae1418c132d4"}
Jul 27 21:28:46.515676 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:46.515532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm" event={"ID":"66024ba2-367a-4408-9381-d62f2641ae76","Type":"ContainerStarted","Data":"5ae0ddea4ff94bcd331a5c89ed579c6564b8be6c71cef3d82fcc7ba1e2ea22c3"}
Jul 27 21:28:47.658331 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:47.658310    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"
Jul 27 21:28:47.691808 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:47.691775    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66024ba2-367a-4408-9381-d62f2641ae76-maas-api-service-ca\") pod \"66024ba2-367a-4408-9381-d62f2641ae76\" (UID: \"66024ba2-367a-4408-9381-d62f2641ae76\") "
Jul 27 21:28:47.692143 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:47.692121    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66024ba2-367a-4408-9381-d62f2641ae76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66024ba2-367a-4408-9381-d62f2641ae76" (UID: "66024ba2-367a-4408-9381-d62f2641ae76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:28:47.793387 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:47.793291    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66024ba2-367a-4408-9381-d62f2641ae76-maas-api-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:48.526486 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:48.526452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm" event={"ID":"66024ba2-367a-4408-9381-d62f2641ae76","Type":"ContainerDied","Data":"5ae0ddea4ff94bcd331a5c89ed579c6564b8be6c71cef3d82fcc7ba1e2ea22c3"}
Jul 27 21:28:48.526486 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:48.526468    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"
Jul 27 21:28:48.526720 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:48.526500    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ae0ddea4ff94bcd331a5c89ed579c6564b8be6c71cef3d82fcc7ba1e2ea22c3"
Jul 27 21:28:55.939853 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:55.939791    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"]
Jul 27 21:28:55.940272 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:55.940075    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8" podUID="2e1daa4e-480b-484c-ad81-718496d68db0" containerName="maas-api" containerID="cri-o://88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d" gracePeriod=30
Jul 27 21:28:55.959437 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:55.959414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:56.038746 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.038714    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"]
Jul 27 21:28:56.038993 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.038967    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz" podUID="e09cbfbe-1811-445c-85a7-0e4a257d92d8" containerName="payload-processing" containerID="cri-o://411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e" gracePeriod=30
Jul 27 21:28:56.117810 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.117777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:56.192834 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.192760    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:28:56.269579 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.269551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8995\" (UniqueName: \"kubernetes.io/projected/2e1daa4e-480b-484c-ad81-718496d68db0-kube-api-access-b8995\") pod \"2e1daa4e-480b-484c-ad81-718496d68db0\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") "
Jul 27 21:28:56.269767 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.269655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1daa4e-480b-484c-ad81-718496d68db0-maas-api-tls\") pod \"2e1daa4e-480b-484c-ad81-718496d68db0\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") "
Jul 27 21:28:56.269767 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.269706    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1daa4e-480b-484c-ad81-718496d68db0-openshift-service-ca\") pod \"2e1daa4e-480b-484c-ad81-718496d68db0\" (UID: \"2e1daa4e-480b-484c-ad81-718496d68db0\") "
Jul 27 21:28:56.270087 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.270058    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e1daa4e-480b-484c-ad81-718496d68db0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e1daa4e-480b-484c-ad81-718496d68db0" (UID: "2e1daa4e-480b-484c-ad81-718496d68db0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:28:56.273517 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.273487    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e1daa4e-480b-484c-ad81-718496d68db0-kube-api-access-b8995" (OuterVolumeSpecName: "kube-api-access-b8995") pod "2e1daa4e-480b-484c-ad81-718496d68db0" (UID: "2e1daa4e-480b-484c-ad81-718496d68db0"). InnerVolumeSpecName "kube-api-access-b8995". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:28:56.273671 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.273646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e1daa4e-480b-484c-ad81-718496d68db0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e1daa4e-480b-484c-ad81-718496d68db0" (UID: "2e1daa4e-480b-484c-ad81-718496d68db0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:28:56.273786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.273763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:28:56.302328 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.302301    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:28:56.371134 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.371101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db4zg\" (UniqueName: \"kubernetes.io/projected/e09cbfbe-1811-445c-85a7-0e4a257d92d8-kube-api-access-db4zg\") pod \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\" (UID: \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\") "
Jul 27 21:28:56.371339 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.371149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09cbfbe-1811-445c-85a7-0e4a257d92d8-plugins-config-volume\") pod \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\" (UID: \"e09cbfbe-1811-445c-85a7-0e4a257d92d8\") "
Jul 27 21:28:56.371428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.371358    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e1daa4e-480b-484c-ad81-718496d68db0-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:56.371428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.371384    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e1daa4e-480b-484c-ad81-718496d68db0-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:56.371428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.371394    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8995\" (UniqueName: \"kubernetes.io/projected/2e1daa4e-480b-484c-ad81-718496d68db0-kube-api-access-b8995\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:56.371592 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.371497    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09cbfbe-1811-445c-85a7-0e4a257d92d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e09cbfbe-1811-445c-85a7-0e4a257d92d8" (UID: "e09cbfbe-1811-445c-85a7-0e4a257d92d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:28:56.373264 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.373242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09cbfbe-1811-445c-85a7-0e4a257d92d8-kube-api-access-db4zg" (OuterVolumeSpecName: "kube-api-access-db4zg") pod "e09cbfbe-1811-445c-85a7-0e4a257d92d8" (UID: "e09cbfbe-1811-445c-85a7-0e4a257d92d8"). InnerVolumeSpecName "kube-api-access-db4zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:28:56.471999 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.471914    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db4zg\" (UniqueName: \"kubernetes.io/projected/e09cbfbe-1811-445c-85a7-0e4a257d92d8-kube-api-access-db4zg\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:56.471999 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.471945    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e09cbfbe-1811-445c-85a7-0e4a257d92d8-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:28:56.566908 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.566868    2577 generic.go:356] "Generic (PLEG): container finished" podID="2e1daa4e-480b-484c-ad81-718496d68db0" containerID="88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d" exitCode=0
Jul 27 21:28:56.567081 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.566929    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"
Jul 27 21:28:56.567081 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.566964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8" event={"ID":"2e1daa4e-480b-484c-ad81-718496d68db0","Type":"ContainerDied","Data":"88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d"}
Jul 27 21:28:56.567081 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.567054    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8" event={"ID":"2e1daa4e-480b-484c-ad81-718496d68db0","Type":"ContainerDied","Data":"8f319854c365104be85a122ad28bbb387af550cc5cae730821818ec39679251b"}
Jul 27 21:28:56.567081 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.567078    2577 scope.go:117] "RemoveContainer" containerID="88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d"
Jul 27 21:28:56.568482 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.568454    2577 generic.go:356] "Generic (PLEG): container finished" podID="e09cbfbe-1811-445c-85a7-0e4a257d92d8" containerID="411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e" exitCode=0
Jul 27 21:28:56.568612 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.568526    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz" event={"ID":"e09cbfbe-1811-445c-85a7-0e4a257d92d8","Type":"ContainerDied","Data":"411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e"}
Jul 27 21:28:56.568612 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.568571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz" event={"ID":"e09cbfbe-1811-445c-85a7-0e4a257d92d8","Type":"ContainerDied","Data":"e5758f8c32ed9968b40eebe8455b2075c4c8d9213d589f88ae0ae07b78ab36a0"}
Jul 27 21:28:56.568723 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.568574    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"
Jul 27 21:28:56.577307 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.577289    2577 scope.go:117] "RemoveContainer" containerID="88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d"
Jul 27 21:28:56.577605 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:28:56.577581    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d\": container with ID starting with 88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d not found: ID does not exist" containerID="88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d"
Jul 27 21:28:56.577713 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.577615    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d"} err="failed to get container status \"88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d\": rpc error: code = NotFound desc = could not find container \"88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d\": container with ID starting with 88eeb2b80dd07e9204ece41ced5788ee0c8f17757e29ceda3dc8475acf46236d not found: ID does not exist"
Jul 27 21:28:56.577713 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.577639    2577 scope.go:117] "RemoveContainer" containerID="411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e"
Jul 27 21:28:56.585923 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.585906    2577 scope.go:117] "RemoveContainer" containerID="411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e"
Jul 27 21:28:56.586196 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:28:56.586161    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e\": container with ID starting with 411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e not found: ID does not exist" containerID="411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e"
Jul 27 21:28:56.586265 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.586246    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e"} err="failed to get container status \"411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e\": rpc error: code = NotFound desc = could not find container \"411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e\": container with ID starting with 411a5637e929bdc74ae1a0decdf3640db4384df6f8fa0925c22390862f2a300e not found: ID does not exist"
Jul 27 21:28:56.605163 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.605134    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"]
Jul 27 21:28:56.611318 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.611294    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0adc64-65cb6cd6-w9tx8"]
Jul 27 21:28:56.631532 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.631504    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"]
Jul 27 21:28:56.634431 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:56.634407    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0adc64-78b6d8bd96-6wfpz"]
Jul 27 21:28:57.184275 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:57.184243    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e1daa4e-480b-484c-ad81-718496d68db0" path="/var/lib/kubelet/pods/2e1daa4e-480b-484c-ad81-718496d68db0/volumes"
Jul 27 21:28:57.184655 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:57.184590    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e09cbfbe-1811-445c-85a7-0e4a257d92d8" path="/var/lib/kubelet/pods/e09cbfbe-1811-445c-85a7-0e4a257d92d8/volumes"
Jul 27 21:28:59.210602 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:28:59.210564    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:29:00.329040 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:00.329000    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"]
Jul 27 21:29:00.334648 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:00.334619    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0adc64-cwgfm"]
Jul 27 21:29:00.379247 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:00.379215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:00.888873 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:00.888837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:01.184442 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:01.184366    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66024ba2-367a-4408-9381-d62f2641ae76" path="/var/lib/kubelet/pods/66024ba2-367a-4408-9381-d62f2641ae76/volumes"
Jul 27 21:29:08.036665 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:08.036626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:08.166114 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:08.166075    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:08.328178 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:08.328060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:08.759552 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:08.759512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:09.017344 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:09.017258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:09.390671 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:09.390564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:09.959670 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:09.959641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:12.231515 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.231478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:12.394555 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.394521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:12.414506 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.414471    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"]
Jul 27 21:29:12.415015 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415001    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66024ba2-367a-4408-9381-d62f2641ae76" containerName="revoke-keys"
Jul 27 21:29:12.415063 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415018    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66024ba2-367a-4408-9381-d62f2641ae76" containerName="revoke-keys"
Jul 27 21:29:12.415063 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415041    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09cbfbe-1811-445c-85a7-0e4a257d92d8" containerName="payload-processing"
Jul 27 21:29:12.415063 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415050    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09cbfbe-1811-445c-85a7-0e4a257d92d8" containerName="payload-processing"
Jul 27 21:29:12.415165 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415064    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e1daa4e-480b-484c-ad81-718496d68db0" containerName="maas-api"
Jul 27 21:29:12.415165 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415072    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e1daa4e-480b-484c-ad81-718496d68db0" containerName="maas-api"
Jul 27 21:29:12.415165 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415161    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e1daa4e-480b-484c-ad81-718496d68db0" containerName="maas-api"
Jul 27 21:29:12.415303 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415197    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e09cbfbe-1811-445c-85a7-0e4a257d92d8" containerName="payload-processing"
Jul 27 21:29:12.415303 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.415209    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66024ba2-367a-4408-9381-d62f2641ae76" containerName="revoke-keys"
Jul 27 21:29:12.419597 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.419580    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.427108 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.427091    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c3ee91-serving-cert\""
Jul 27 21:29:12.427537 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.427523    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:29:12.431248 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.428104    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cgjq6\""
Jul 27 21:29:12.431248 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.429823    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:29:12.462160 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.462124    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"]
Jul 27 21:29:12.517578 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.517492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfd8m\" (UniqueName: \"kubernetes.io/projected/bfdb8670-ec02-4839-aa50-4426face7356-kube-api-access-pfd8m\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.517578 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.517555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdb8670-ec02-4839-aa50-4426face7356-maas-api-tls\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.517767 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.517649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdb8670-ec02-4839-aa50-4426face7356-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.574985 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.574941    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:12.618950 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.618919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdb8670-ec02-4839-aa50-4426face7356-maas-api-tls\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.619073 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.618973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdb8670-ec02-4839-aa50-4426face7356-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.619073 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.619015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfd8m\" (UniqueName: \"kubernetes.io/projected/bfdb8670-ec02-4839-aa50-4426face7356-kube-api-access-pfd8m\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.619711 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.619690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdb8670-ec02-4839-aa50-4426face7356-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.621531 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.621512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdb8670-ec02-4839-aa50-4426face7356-maas-api-tls\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.635150 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.635123    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfd8m\" (UniqueName: \"kubernetes.io/projected/bfdb8670-ec02-4839-aa50-4426face7356-kube-api-access-pfd8m\") pod \"maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.730174 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.730136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:12.816057 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.815916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:12.902796 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:12.902772    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"]
Jul 27 21:29:12.904815 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:29:12.904784    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdb8670_ec02_4839_aa50_4426face7356.slice/crio-86c1542e10752343baa17e7215e82484cf35217ee6fe69cb4d2ce79fb13f73c4 WatchSource:0}: Error finding container 86c1542e10752343baa17e7215e82484cf35217ee6fe69cb4d2ce79fb13f73c4: Status 404 returned error can't find the container with id 86c1542e10752343baa17e7215e82484cf35217ee6fe69cb4d2ce79fb13f73c4
Jul 27 21:29:13.416022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.415984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:13.516796 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.516770    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:13.639035 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.639005    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"]
Jul 27 21:29:13.642314 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.642298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:13.645584 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.645565    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c3ee91-dockercfg-jfdn6\""
Jul 27 21:29:13.645666 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.645636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c3ee91\""
Jul 27 21:29:13.653131 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.653102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn" event={"ID":"bfdb8670-ec02-4839-aa50-4426face7356","Type":"ContainerStarted","Data":"a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c"}
Jul 27 21:29:13.653262 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.653168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:13.653262 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.653222    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn" event={"ID":"bfdb8670-ec02-4839-aa50-4426face7356","Type":"ContainerStarted","Data":"86c1542e10752343baa17e7215e82484cf35217ee6fe69cb4d2ce79fb13f73c4"}
Jul 27 21:29:13.655155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.655136    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"]
Jul 27 21:29:13.700487 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.700382    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn" podStartSLOduration=1.1253494179999999 podStartE2EDuration="1.70036291s" totalImagesPullingTime="575.013492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:13.699606403 +0000 UTC m=+2271.065172249" watchObservedRunningTime="2026-07-27 21:29:13.70036291 +0000 UTC m=+2271.065928737"
Jul 27 21:29:13.713276 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.713254    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"]
Jul 27 21:29:13.716568 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.716550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:13.729595 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.729570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"]
Jul 27 21:29:13.830374 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.830335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ea389-60a2-483c-9c59-fe874db16503-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z\" (UID: \"6e7ea389-60a2-483c-9c59-fe874db16503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:13.830374 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.830376    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13956ddd-c727-4e2b-9160-787f0ed85ba9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6\" (UID: \"13956ddd-c727-4e2b-9160-787f0ed85ba9\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:13.830606 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.830482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d64b\" (UniqueName: \"kubernetes.io/projected/13956ddd-c727-4e2b-9160-787f0ed85ba9-kube-api-access-8d64b\") pod \"payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6\" (UID: \"13956ddd-c727-4e2b-9160-787f0ed85ba9\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:13.830606 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.830529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f9v\" (UniqueName: \"kubernetes.io/projected/6e7ea389-60a2-483c-9c59-fe874db16503-kube-api-access-b6f9v\") pod \"payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z\" (UID: \"6e7ea389-60a2-483c-9c59-fe874db16503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:13.931546 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.931512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ea389-60a2-483c-9c59-fe874db16503-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z\" (UID: \"6e7ea389-60a2-483c-9c59-fe874db16503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:13.931546 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.931550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13956ddd-c727-4e2b-9160-787f0ed85ba9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6\" (UID: \"13956ddd-c727-4e2b-9160-787f0ed85ba9\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:13.931793 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.931595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d64b\" (UniqueName: \"kubernetes.io/projected/13956ddd-c727-4e2b-9160-787f0ed85ba9-kube-api-access-8d64b\") pod \"payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6\" (UID: \"13956ddd-c727-4e2b-9160-787f0ed85ba9\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:13.931793 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.931617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f9v\" (UniqueName: \"kubernetes.io/projected/6e7ea389-60a2-483c-9c59-fe874db16503-kube-api-access-b6f9v\") pod \"payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z\" (UID: \"6e7ea389-60a2-483c-9c59-fe874db16503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:13.932460 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.932439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13956ddd-c727-4e2b-9160-787f0ed85ba9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6\" (UID: \"13956ddd-c727-4e2b-9160-787f0ed85ba9\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:13.932519 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.932439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ea389-60a2-483c-9c59-fe874db16503-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z\" (UID: \"6e7ea389-60a2-483c-9c59-fe874db16503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:13.942262 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.942227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f9v\" (UniqueName: \"kubernetes.io/projected/6e7ea389-60a2-483c-9c59-fe874db16503-kube-api-access-b6f9v\") pod \"payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z\" (UID: \"6e7ea389-60a2-483c-9c59-fe874db16503\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:13.943258 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.943240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d64b\" (UniqueName: \"kubernetes.io/projected/13956ddd-c727-4e2b-9160-787f0ed85ba9-kube-api-access-8d64b\") pod \"payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6\" (UID: \"13956ddd-c727-4e2b-9160-787f0ed85ba9\") " pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:13.954197 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:13.954121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:14.025673 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.025638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:14.123913 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.123879    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"]
Jul 27 21:29:14.126373 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:29:14.126343    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7ea389_60a2_483c_9c59_fe874db16503.slice/crio-3240c6745d0328ee00f961b432a593e831115f66952cf80f3fab43cc57741c8d WatchSource:0}: Error finding container 3240c6745d0328ee00f961b432a593e831115f66952cf80f3fab43cc57741c8d: Status 404 returned error can't find the container with id 3240c6745d0328ee00f961b432a593e831115f66952cf80f3fab43cc57741c8d
Jul 27 21:29:14.162869 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.162842    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"]
Jul 27 21:29:14.166069 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:29:14.166047    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13956ddd_c727_4e2b_9160_787f0ed85ba9.slice/crio-72a158348ab103ac59667ba59da55259bdb50d9e496298c9b8d7d8f73ab113d3 WatchSource:0}: Error finding container 72a158348ab103ac59667ba59da55259bdb50d9e496298c9b8d7d8f73ab113d3: Status 404 returned error can't find the container with id 72a158348ab103ac59667ba59da55259bdb50d9e496298c9b8d7d8f73ab113d3
Jul 27 21:29:14.660801 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.660761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z" event={"ID":"6e7ea389-60a2-483c-9c59-fe874db16503","Type":"ContainerStarted","Data":"34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9"}
Jul 27 21:29:14.661286 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.660819    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z" event={"ID":"6e7ea389-60a2-483c-9c59-fe874db16503","Type":"ContainerStarted","Data":"3240c6745d0328ee00f961b432a593e831115f66952cf80f3fab43cc57741c8d"}
Jul 27 21:29:14.661286 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.660891    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:14.662492 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.662465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6" event={"ID":"13956ddd-c727-4e2b-9160-787f0ed85ba9","Type":"ContainerStarted","Data":"0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a"}
Jul 27 21:29:14.662607 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.662514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6" event={"ID":"13956ddd-c727-4e2b-9160-787f0ed85ba9","Type":"ContainerStarted","Data":"72a158348ab103ac59667ba59da55259bdb50d9e496298c9b8d7d8f73ab113d3"}
Jul 27 21:29:14.662693 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.662679    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:14.684115 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.684074    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z" podStartSLOduration=1.684062022 podStartE2EDuration="1.684062022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:14.682229473 +0000 UTC m=+2272.047795314" watchObservedRunningTime="2026-07-27 21:29:14.684062022 +0000 UTC m=+2272.049627869"
Jul 27 21:29:14.710373 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:14.710330    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6" podStartSLOduration=1.710318852 podStartE2EDuration="1.710318852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:14.709039461 +0000 UTC m=+2272.074605308" watchObservedRunningTime="2026-07-27 21:29:14.710318852 +0000 UTC m=+2272.075884698"
Jul 27 21:29:15.378888 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:15.378850    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:29:16.578280 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:16.578178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:17.155681 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:17.155643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:17.628575 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:17.628538    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:17.775461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:17.775413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:19.668777 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:19.668749    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:20.299533 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.299503    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:20.490082 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.490047    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:20.528273 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.528239    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"]
Jul 27 21:29:20.532520 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.532494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.538480 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.538454    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-86cc77-serving-cert\""
Jul 27 21:29:20.556950 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.556875    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"]
Jul 27 21:29:20.588292 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.588242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.588292 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.588294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8767f7d1-44d0-4300-8d68-7994a5fc45b2-openshift-service-ca\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.588534 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.588372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8fjj\" (UniqueName: \"kubernetes.io/projected/8767f7d1-44d0-4300-8d68-7994a5fc45b2-kube-api-access-v8fjj\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.689261 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.689228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.689261 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.689266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8767f7d1-44d0-4300-8d68-7994a5fc45b2-openshift-service-ca\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.689789 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.689303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8fjj\" (UniqueName: \"kubernetes.io/projected/8767f7d1-44d0-4300-8d68-7994a5fc45b2-kube-api-access-v8fjj\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.689789 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:29:20.689409    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-serving-cert: secret "maas-api-e2e-api-b-86cc77-serving-cert" not found
Jul 27 21:29:20.689789 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:29:20.689504    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls podName:8767f7d1-44d0-4300-8d68-7994a5fc45b2 nodeName:}" failed. No retries permitted until 2026-07-27 21:29:21.189482322 +0000 UTC m=+2278.555048152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls") pod "maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88" (UID: "8767f7d1-44d0-4300-8d68-7994a5fc45b2") : secret "maas-api-e2e-api-b-86cc77-serving-cert" not found
Jul 27 21:29:20.689906 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.689865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8767f7d1-44d0-4300-8d68-7994a5fc45b2-openshift-service-ca\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.720164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.720135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8fjj\" (UniqueName: \"kubernetes.io/projected/8767f7d1-44d0-4300-8d68-7994a5fc45b2-kube-api-access-v8fjj\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:20.861429 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:20.861344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:21.184241 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:21.184107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:21.193595 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:21.193566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:21.196136 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:21.196112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls\") pod \"maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:21.445850 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:21.445763    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:21.617473 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:21.617340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"]
Jul 27 21:29:21.620395 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:29:21.620369    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8767f7d1_44d0_4300_8d68_7994a5fc45b2.slice/crio-44656605c972b71e927ff4fa289c8289fa03b35ee487bf17433fcc444ee6f0f6 WatchSource:0}: Error finding container 44656605c972b71e927ff4fa289c8289fa03b35ee487bf17433fcc444ee6f0f6: Status 404 returned error can't find the container with id 44656605c972b71e927ff4fa289c8289fa03b35ee487bf17433fcc444ee6f0f6
Jul 27 21:29:21.701040 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:21.700957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88" event={"ID":"8767f7d1-44d0-4300-8d68-7994a5fc45b2","Type":"ContainerStarted","Data":"44656605c972b71e927ff4fa289c8289fa03b35ee487bf17433fcc444ee6f0f6"}
Jul 27 21:29:22.016725 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.016686    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:22.163504 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.163470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:22.301332 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.301241    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"]
Jul 27 21:29:22.304877 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.304855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:22.307549 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.307531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-86cc77\""
Jul 27 21:29:22.307650 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.307548    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-86cc77-dockercfg-xkkvq\""
Jul 27 21:29:22.332099 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.332063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"]
Jul 27 21:29:22.367851 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.367810    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"]
Jul 27 21:29:22.371403 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.371381    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:22.395325 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.395297    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"]
Jul 27 21:29:22.406443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.406415    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acb5cf7-b965-4501-948b-49e3f93f155a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk\" (UID: \"7acb5cf7-b965-4501-948b-49e3f93f155a\") " pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:22.406596 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.406451    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlxwc\" (UniqueName: \"kubernetes.io/projected/7acb5cf7-b965-4501-948b-49e3f93f155a-kube-api-access-rlxwc\") pod \"payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk\" (UID: \"7acb5cf7-b965-4501-948b-49e3f93f155a\") " pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:22.406596 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.406480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9546s\" (UniqueName: \"kubernetes.io/projected/ed6cec75-5242-43e8-b854-18100696d4de-kube-api-access-9546s\") pod \"payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d\" (UID: \"ed6cec75-5242-43e8-b854-18100696d4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:22.406596 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.406566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6cec75-5242-43e8-b854-18100696d4de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d\" (UID: \"ed6cec75-5242-43e8-b854-18100696d4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:22.507922 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.507885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6cec75-5242-43e8-b854-18100696d4de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d\" (UID: \"ed6cec75-5242-43e8-b854-18100696d4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:22.508093 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.507982    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acb5cf7-b965-4501-948b-49e3f93f155a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk\" (UID: \"7acb5cf7-b965-4501-948b-49e3f93f155a\") " pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:22.508093 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.508012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlxwc\" (UniqueName: \"kubernetes.io/projected/7acb5cf7-b965-4501-948b-49e3f93f155a-kube-api-access-rlxwc\") pod \"payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk\" (UID: \"7acb5cf7-b965-4501-948b-49e3f93f155a\") " pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:22.508093 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.508051    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9546s\" (UniqueName: \"kubernetes.io/projected/ed6cec75-5242-43e8-b854-18100696d4de-kube-api-access-9546s\") pod \"payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d\" (UID: \"ed6cec75-5242-43e8-b854-18100696d4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:22.508718 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.508698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6cec75-5242-43e8-b854-18100696d4de-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d\" (UID: \"ed6cec75-5242-43e8-b854-18100696d4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:22.508764 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.508714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acb5cf7-b965-4501-948b-49e3f93f155a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk\" (UID: \"7acb5cf7-b965-4501-948b-49e3f93f155a\") " pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:22.519012 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.518987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9546s\" (UniqueName: \"kubernetes.io/projected/ed6cec75-5242-43e8-b854-18100696d4de-kube-api-access-9546s\") pod \"payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d\" (UID: \"ed6cec75-5242-43e8-b854-18100696d4de\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:22.519218 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.519078    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlxwc\" (UniqueName: \"kubernetes.io/projected/7acb5cf7-b965-4501-948b-49e3f93f155a-kube-api-access-rlxwc\") pod \"payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk\" (UID: \"7acb5cf7-b965-4501-948b-49e3f93f155a\") " pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:22.615037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.614948    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:22.681742 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.681713    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:22.709678 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.709610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88" event={"ID":"8767f7d1-44d0-4300-8d68-7994a5fc45b2","Type":"ContainerStarted","Data":"3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef"}
Jul 27 21:29:22.710107 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.709767    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:22.792206 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.790958    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88" podStartSLOduration=2.281853246 podStartE2EDuration="2.790933551s" totalImagesPullingTime="509.080305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:22.765787885 +0000 UTC m=+2280.131353733" watchObservedRunningTime="2026-07-27 21:29:22.790933551 +0000 UTC m=+2280.156499398"
Jul 27 21:29:22.792723 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:22.792696    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"]
Jul 27 21:29:22.794171 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:29:22.794142    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acb5cf7_b965_4501_948b_49e3f93f155a.slice/crio-656ce03c20ff793548800bdb4d4bc118e058925758a1dfe8b9af13809d8e0e09 WatchSource:0}: Error finding container 656ce03c20ff793548800bdb4d4bc118e058925758a1dfe8b9af13809d8e0e09: Status 404 returned error can't find the container with id 656ce03c20ff793548800bdb4d4bc118e058925758a1dfe8b9af13809d8e0e09
Jul 27 21:29:23.052495 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.052462    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"]
Jul 27 21:29:23.059639 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:29:23.059611    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6cec75_5242_43e8_b854_18100696d4de.slice/crio-20796fccf239a105ca23c69a17f062ac757a351faff4b0e43427eeef2c75f0d7 WatchSource:0}: Error finding container 20796fccf239a105ca23c69a17f062ac757a351faff4b0e43427eeef2c75f0d7: Status 404 returned error can't find the container with id 20796fccf239a105ca23c69a17f062ac757a351faff4b0e43427eeef2c75f0d7
Jul 27 21:29:23.376777 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.376695    2577 scope.go:117] "RemoveContainer" containerID="91556179cd86b6cae4ccad1fd690b8b7e108f742941b0e5efc3bbd5fe55eeacc"
Jul 27 21:29:23.715917 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.715828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d" event={"ID":"ed6cec75-5242-43e8-b854-18100696d4de","Type":"ContainerStarted","Data":"28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea"}
Jul 27 21:29:23.715917 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.715872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d" event={"ID":"ed6cec75-5242-43e8-b854-18100696d4de","Type":"ContainerStarted","Data":"20796fccf239a105ca23c69a17f062ac757a351faff4b0e43427eeef2c75f0d7"}
Jul 27 21:29:23.715917 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.715901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:23.717541 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.717517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk" event={"ID":"7acb5cf7-b965-4501-948b-49e3f93f155a","Type":"ContainerStarted","Data":"945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00"}
Jul 27 21:29:23.717608 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.717564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk" event={"ID":"7acb5cf7-b965-4501-948b-49e3f93f155a","Type":"ContainerStarted","Data":"656ce03c20ff793548800bdb4d4bc118e058925758a1dfe8b9af13809d8e0e09"}
Jul 27 21:29:23.717608 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.717589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:23.764223 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.764155    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d" podStartSLOduration=1.764137726 podStartE2EDuration="1.764137726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:23.762779823 +0000 UTC m=+2281.128345673" watchObservedRunningTime="2026-07-27 21:29:23.764137726 +0000 UTC m=+2281.129703572"
Jul 27 21:29:23.809313 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:23.809255    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk" podStartSLOduration=1.809240416 podStartE2EDuration="1.809240416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:23.808392295 +0000 UTC m=+2281.173958142" watchObservedRunningTime="2026-07-27 21:29:23.809240416 +0000 UTC m=+2281.174806263"
Jul 27 21:29:24.819298 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:24.819256    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:25.669416 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:25.669382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:25.669614 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:25.669596    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:28.693807 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:28.693767    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:29:28.723045 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:28.723019    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:29:34.724462 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:34.724424    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:29:34.724859 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:34.724574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:29:48.347252 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.347218    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"]
Jul 27 21:29:48.347823 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.347495    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn" podUID="bfdb8670-ec02-4839-aa50-4426face7356" containerName="maas-api" containerID="cri-o://a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c" gracePeriod=30
Jul 27 21:29:48.421149 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.421111    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:48.476360 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.476328    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"]
Jul 27 21:29:48.476589 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.476550    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6" podUID="13956ddd-c727-4e2b-9160-787f0ed85ba9" containerName="payload-processing" containerID="cri-o://0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a" gracePeriod=30
Jul 27 21:29:48.490103 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.490063    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"]
Jul 27 21:29:48.490410 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.490361    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z" podUID="6e7ea389-60a2-483c-9c59-fe874db16503" containerName="payload-pre-processing" containerID="cri-o://34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9" gracePeriod=30
Jul 27 21:29:48.532076 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.532036    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:48.616076 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.616052    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:48.663375 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.663335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:48.745549 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.745517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfd8m\" (UniqueName: \"kubernetes.io/projected/bfdb8670-ec02-4839-aa50-4426face7356-kube-api-access-pfd8m\") pod \"bfdb8670-ec02-4839-aa50-4426face7356\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") "
Jul 27 21:29:48.745728 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.745612    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdb8670-ec02-4839-aa50-4426face7356-openshift-service-ca\") pod \"bfdb8670-ec02-4839-aa50-4426face7356\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") "
Jul 27 21:29:48.745728 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.745656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdb8670-ec02-4839-aa50-4426face7356-maas-api-tls\") pod \"bfdb8670-ec02-4839-aa50-4426face7356\" (UID: \"bfdb8670-ec02-4839-aa50-4426face7356\") "
Jul 27 21:29:48.745991 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.745962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdb8670-ec02-4839-aa50-4426face7356-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bfdb8670-ec02-4839-aa50-4426face7356" (UID: "bfdb8670-ec02-4839-aa50-4426face7356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:48.748710 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.748686    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdb8670-ec02-4839-aa50-4426face7356-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bfdb8670-ec02-4839-aa50-4426face7356" (UID: "bfdb8670-ec02-4839-aa50-4426face7356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:29:48.748798 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.748687    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdb8670-ec02-4839-aa50-4426face7356-kube-api-access-pfd8m" (OuterVolumeSpecName: "kube-api-access-pfd8m") pod "bfdb8670-ec02-4839-aa50-4426face7356" (UID: "bfdb8670-ec02-4839-aa50-4426face7356"). InnerVolumeSpecName "kube-api-access-pfd8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:48.756799 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.756774    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:48.765441 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.765422    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:48.843547 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.843508    2577 generic.go:356] "Generic (PLEG): container finished" podID="13956ddd-c727-4e2b-9160-787f0ed85ba9" containerID="0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a" exitCode=0
Jul 27 21:29:48.843547 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.843553    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"
Jul 27 21:29:48.843770 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.843595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6" event={"ID":"13956ddd-c727-4e2b-9160-787f0ed85ba9","Type":"ContainerDied","Data":"0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a"}
Jul 27 21:29:48.843770 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.843651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6" event={"ID":"13956ddd-c727-4e2b-9160-787f0ed85ba9","Type":"ContainerDied","Data":"72a158348ab103ac59667ba59da55259bdb50d9e496298c9b8d7d8f73ab113d3"}
Jul 27 21:29:48.843770 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.843674    2577 scope.go:117] "RemoveContainer" containerID="0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a"
Jul 27 21:29:48.844925 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.844906    2577 generic.go:356] "Generic (PLEG): container finished" podID="bfdb8670-ec02-4839-aa50-4426face7356" containerID="a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c" exitCode=0
Jul 27 21:29:48.845035 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.844946    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"
Jul 27 21:29:48.845035 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.844976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn" event={"ID":"bfdb8670-ec02-4839-aa50-4426face7356","Type":"ContainerDied","Data":"a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c"}
Jul 27 21:29:48.845035 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.845013    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn" event={"ID":"bfdb8670-ec02-4839-aa50-4426face7356","Type":"ContainerDied","Data":"86c1542e10752343baa17e7215e82484cf35217ee6fe69cb4d2ce79fb13f73c4"}
Jul 27 21:29:48.846317 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13956ddd-c727-4e2b-9160-787f0ed85ba9-plugins-config-volume\") pod \"13956ddd-c727-4e2b-9160-787f0ed85ba9\" (UID: \"13956ddd-c727-4e2b-9160-787f0ed85ba9\") "
Jul 27 21:29:48.846452 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846352    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ea389-60a2-483c-9c59-fe874db16503-plugins-config-volume\") pod \"6e7ea389-60a2-483c-9c59-fe874db16503\" (UID: \"6e7ea389-60a2-483c-9c59-fe874db16503\") "
Jul 27 21:29:48.846452 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846385    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6f9v\" (UniqueName: \"kubernetes.io/projected/6e7ea389-60a2-483c-9c59-fe874db16503-kube-api-access-b6f9v\") pod \"6e7ea389-60a2-483c-9c59-fe874db16503\" (UID: \"6e7ea389-60a2-483c-9c59-fe874db16503\") "
Jul 27 21:29:48.846452 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846403    2577 generic.go:356] "Generic (PLEG): container finished" podID="6e7ea389-60a2-483c-9c59-fe874db16503" containerID="34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9" exitCode=0
Jul 27 21:29:48.846616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846448    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z" event={"ID":"6e7ea389-60a2-483c-9c59-fe874db16503","Type":"ContainerDied","Data":"34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9"}
Jul 27 21:29:48.846616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846480    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d64b\" (UniqueName: \"kubernetes.io/projected/13956ddd-c727-4e2b-9160-787f0ed85ba9-kube-api-access-8d64b\") pod \"13956ddd-c727-4e2b-9160-787f0ed85ba9\" (UID: \"13956ddd-c727-4e2b-9160-787f0ed85ba9\") "
Jul 27 21:29:48.846616 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z" event={"ID":"6e7ea389-60a2-483c-9c59-fe874db16503","Type":"ContainerDied","Data":"3240c6745d0328ee00f961b432a593e831115f66952cf80f3fab43cc57741c8d"}
Jul 27 21:29:48.846779 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846656    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13956ddd-c727-4e2b-9160-787f0ed85ba9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "13956ddd-c727-4e2b-9160-787f0ed85ba9" (UID: "13956ddd-c727-4e2b-9160-787f0ed85ba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:48.846779 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7ea389-60a2-483c-9c59-fe874db16503-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e7ea389-60a2-483c-9c59-fe874db16503" (UID: "6e7ea389-60a2-483c-9c59-fe874db16503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:48.846779 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846742    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13956ddd-c727-4e2b-9160-787f0ed85ba9-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:29:48.846779 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846760    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdb8670-ec02-4839-aa50-4426face7356-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:29:48.846779 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846771    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdb8670-ec02-4839-aa50-4426face7356-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:29:48.847027 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846785    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfd8m\" (UniqueName: \"kubernetes.io/projected/bfdb8670-ec02-4839-aa50-4426face7356-kube-api-access-pfd8m\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:29:48.847027 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.846831    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"
Jul 27 21:29:48.848887 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.848820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7ea389-60a2-483c-9c59-fe874db16503-kube-api-access-b6f9v" (OuterVolumeSpecName: "kube-api-access-b6f9v") pod "6e7ea389-60a2-483c-9c59-fe874db16503" (UID: "6e7ea389-60a2-483c-9c59-fe874db16503"). InnerVolumeSpecName "kube-api-access-b6f9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:48.849264 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.849161    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13956ddd-c727-4e2b-9160-787f0ed85ba9-kube-api-access-8d64b" (OuterVolumeSpecName: "kube-api-access-8d64b") pod "13956ddd-c727-4e2b-9160-787f0ed85ba9" (UID: "13956ddd-c727-4e2b-9160-787f0ed85ba9"). InnerVolumeSpecName "kube-api-access-8d64b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:48.853416 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.853391    2577 scope.go:117] "RemoveContainer" containerID="0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a"
Jul 27 21:29:48.853747 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:29:48.853713    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a\": container with ID starting with 0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a not found: ID does not exist" containerID="0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a"
Jul 27 21:29:48.853840 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.853760    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a"} err="failed to get container status \"0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a\": rpc error: code = NotFound desc = could not find container \"0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a\": container with ID starting with 0a128f030d89cef7f768f52579ca8e5404df414691a3209644413f872f0c3c7a not found: ID does not exist"
Jul 27 21:29:48.853840 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.853783    2577 scope.go:117] "RemoveContainer" containerID="a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c"
Jul 27 21:29:48.861623 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.861608    2577 scope.go:117] "RemoveContainer" containerID="a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c"
Jul 27 21:29:48.861877 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:29:48.861860    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c\": container with ID starting with a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c not found: ID does not exist" containerID="a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c"
Jul 27 21:29:48.861930 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.861885    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c"} err="failed to get container status \"a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c\": rpc error: code = NotFound desc = could not find container \"a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c\": container with ID starting with a7df9bd0912eb138b82a283146a3babd5c7694f819442197058b2c724583264c not found: ID does not exist"
Jul 27 21:29:48.861930 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.861901    2577 scope.go:117] "RemoveContainer" containerID="34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9"
Jul 27 21:29:48.871681 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.871626    2577 scope.go:117] "RemoveContainer" containerID="34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9"
Jul 27 21:29:48.872689 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:29:48.872655    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9\": container with ID starting with 34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9 not found: ID does not exist" containerID="34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9"
Jul 27 21:29:48.872786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.872698    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9"} err="failed to get container status \"34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9\": rpc error: code = NotFound desc = could not find container \"34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9\": container with ID starting with 34a6e9e94014b368b96dafc668d72c30535a536d7d1fdb521b8fd03d061e73b9 not found: ID does not exist"
Jul 27 21:29:48.872955 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.872941    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"]
Jul 27 21:29:48.874442 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.874424    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3ee91-6557867c5d-hzbvn"]
Jul 27 21:29:48.947776 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.947747    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d64b\" (UniqueName: \"kubernetes.io/projected/13956ddd-c727-4e2b-9160-787f0ed85ba9-kube-api-access-8d64b\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:29:48.947776 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.947772    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7ea389-60a2-483c-9c59-fe874db16503-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:29:48.947776 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:48.947782    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6f9v\" (UniqueName: \"kubernetes.io/projected/6e7ea389-60a2-483c-9c59-fe874db16503-kube-api-access-b6f9v\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:29:49.166653 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:49.166622    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"]
Jul 27 21:29:49.171047 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:49.171024    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c3ee91-5755cdd8bd-6dlp6"]
Jul 27 21:29:49.185324 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:49.185299    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13956ddd-c727-4e2b-9160-787f0ed85ba9" path="/var/lib/kubelet/pods/13956ddd-c727-4e2b-9160-787f0ed85ba9/volumes"
Jul 27 21:29:49.185708 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:49.185694    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfdb8670-ec02-4839-aa50-4426face7356" path="/var/lib/kubelet/pods/bfdb8670-ec02-4839-aa50-4426face7356/volumes"
Jul 27 21:29:49.185933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:49.185922    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"]
Jul 27 21:29:49.186807 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:49.186787    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c3ee91-847885cf56-phr8z"]
Jul 27 21:29:49.971055 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:49.971009    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:29:50.134991 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:50.134948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:51.185273 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:51.185243    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e7ea389-60a2-483c-9c59-fe874db16503" path="/var/lib/kubelet/pods/6e7ea389-60a2-483c-9c59-fe874db16503/volumes"
Jul 27 21:29:53.090438 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:53.090405    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:53.531407 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:53.531369    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:59.721289 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:59.721237    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:29:59.822709 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:29:59.822673    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:00.006481 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.006439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:00.145202 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145157    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"]
Jul 27 21:30:00.145645 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145626    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdb8670-ec02-4839-aa50-4426face7356" containerName="maas-api"
Jul 27 21:30:00.145736 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145647    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdb8670-ec02-4839-aa50-4426face7356" containerName="maas-api"
Jul 27 21:30:00.145736 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145661    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e7ea389-60a2-483c-9c59-fe874db16503" containerName="payload-pre-processing"
Jul 27 21:30:00.145736 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145669    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e7ea389-60a2-483c-9c59-fe874db16503" containerName="payload-pre-processing"
Jul 27 21:30:00.145736 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145677    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13956ddd-c727-4e2b-9160-787f0ed85ba9" containerName="payload-processing"
Jul 27 21:30:00.145736 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145686    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="13956ddd-c727-4e2b-9160-787f0ed85ba9" containerName="payload-processing"
Jul 27 21:30:00.145993 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145780    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e7ea389-60a2-483c-9c59-fe874db16503" containerName="payload-pre-processing"
Jul 27 21:30:00.145993 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145798    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdb8670-ec02-4839-aa50-4426face7356" containerName="maas-api"
Jul 27 21:30:00.145993 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.145808    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="13956ddd-c727-4e2b-9160-787f0ed85ba9" containerName="payload-processing"
Jul 27 21:30:00.148923 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.148903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"
Jul 27 21:30:00.150854 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.150832    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg"]
Jul 27 21:30:00.151994 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.151977    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fvvt\""
Jul 27 21:30:00.154052 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.154034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg"
Jul 27 21:30:00.159378 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.159344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"
Jul 27 21:30:00.162797 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.162768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"]
Jul 27 21:30:00.167158 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.167141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg"
Jul 27 21:30:00.171174 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.171155    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg"]
Jul 27 21:30:00.244352 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.244271    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:00.314994 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.314963    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg"]
Jul 27 21:30:00.317854 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:30:00.317825    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae0274c_74c9_43ec_bedb_8915a04ee87f.slice/crio-666f94def8f7e5ee2ac9fc793829ff3d42a1f8a0ab0b6317ed249e4bd6f8beaf WatchSource:0}: Error finding container 666f94def8f7e5ee2ac9fc793829ff3d42a1f8a0ab0b6317ed249e4bd6f8beaf: Status 404 returned error can't find the container with id 666f94def8f7e5ee2ac9fc793829ff3d42a1f8a0ab0b6317ed249e4bd6f8beaf
Jul 27 21:30:00.337232 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.337206    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"]
Jul 27 21:30:00.340228 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:30:00.340176    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1ab76f_9286_4e76_88a5_eb75c6ae4b5d.slice/crio-e940234a04b3b26cf5481339ce6f43a1d0129d4a452b8c46d16d4128ff336282 WatchSource:0}: Error finding container e940234a04b3b26cf5481339ce6f43a1d0129d4a452b8c46d16d4128ff336282: Status 404 returned error can't find the container with id e940234a04b3b26cf5481339ce6f43a1d0129d4a452b8c46d16d4128ff336282
Jul 27 21:30:00.904845 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.904736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg" event={"ID":"9ae0274c-74c9-43ec-bedb-8915a04ee87f","Type":"ContainerStarted","Data":"4be1901db305b3c995ab011648c1e0d54cb64c0743dd6a24f3f7ff38446b0be4"}
Jul 27 21:30:00.904845 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.904801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg" event={"ID":"9ae0274c-74c9-43ec-bedb-8915a04ee87f","Type":"ContainerStarted","Data":"666f94def8f7e5ee2ac9fc793829ff3d42a1f8a0ab0b6317ed249e4bd6f8beaf"}
Jul 27 21:30:00.906409 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.906385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62" event={"ID":"ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d","Type":"ContainerStarted","Data":"73abcbae254137956b6a48ee965d7962856e12d7bb3671f78f1c362ee97c9361"}
Jul 27 21:30:00.906527 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.906431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62" event={"ID":"ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d","Type":"ContainerStarted","Data":"e940234a04b3b26cf5481339ce6f43a1d0129d4a452b8c46d16d4128ff336282"}
Jul 27 21:30:00.927126 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.927080    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg" podStartSLOduration=0.927063322 podStartE2EDuration="927.063322ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:00.924916045 +0000 UTC m=+2318.290481893" watchObservedRunningTime="2026-07-27 21:30:00.927063322 +0000 UTC m=+2318.292629169"
Jul 27 21:30:00.944020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:00.943968    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62" podStartSLOduration=0.943954883 podStartE2EDuration="943.954883ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:00.943754023 +0000 UTC m=+2318.309319871" watchObservedRunningTime="2026-07-27 21:30:00.943954883 +0000 UTC m=+2318.309520863"
Jul 27 21:30:01.913112 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:01.913076    2577 generic.go:356] "Generic (PLEG): container finished" podID="ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d" containerID="73abcbae254137956b6a48ee965d7962856e12d7bb3671f78f1c362ee97c9361" exitCode=0
Jul 27 21:30:01.913561 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:01.913168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62" event={"ID":"ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d","Type":"ContainerDied","Data":"73abcbae254137956b6a48ee965d7962856e12d7bb3671f78f1c362ee97c9361"}
Jul 27 21:30:01.914806 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:01.914783    2577 generic.go:356] "Generic (PLEG): container finished" podID="9ae0274c-74c9-43ec-bedb-8915a04ee87f" containerID="4be1901db305b3c995ab011648c1e0d54cb64c0743dd6a24f3f7ff38446b0be4" exitCode=0
Jul 27 21:30:01.914923 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:01.914816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg" event={"ID":"9ae0274c-74c9-43ec-bedb-8915a04ee87f","Type":"ContainerDied","Data":"4be1901db305b3c995ab011648c1e0d54cb64c0743dd6a24f3f7ff38446b0be4"}
Jul 27 21:30:03.085362 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:03.085340    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg"
Jul 27 21:30:03.088710 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:03.088691    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"
Jul 27 21:30:03.925672 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:03.925641    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"
Jul 27 21:30:03.925871 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:03.925641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62" event={"ID":"ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d","Type":"ContainerDied","Data":"e940234a04b3b26cf5481339ce6f43a1d0129d4a452b8c46d16d4128ff336282"}
Jul 27 21:30:03.925871 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:03.925779    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e940234a04b3b26cf5481339ce6f43a1d0129d4a452b8c46d16d4128ff336282"
Jul 27 21:30:03.927051 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:03.927029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg" event={"ID":"9ae0274c-74c9-43ec-bedb-8915a04ee87f","Type":"ContainerDied","Data":"666f94def8f7e5ee2ac9fc793829ff3d42a1f8a0ab0b6317ed249e4bd6f8beaf"}
Jul 27 21:30:03.927168 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:03.927065    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666f94def8f7e5ee2ac9fc793829ff3d42a1f8a0ab0b6317ed249e4bd6f8beaf"
Jul 27 21:30:03.927168 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:03.927038    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-v2szg"
Jul 27 21:30:16.334541 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.334446    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"]
Jul 27 21:30:16.335001 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.334699    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88" podUID="8767f7d1-44d0-4300-8d68-7994a5fc45b2" containerName="maas-api" containerID="cri-o://3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef" gracePeriod=30
Jul 27 21:30:16.366955 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.366914    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"]
Jul 27 21:30:16.369257 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.369207    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-86cc77-29753130-46d62"]
Jul 27 21:30:16.370448 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.370423    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:16.408512 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.408471    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"]
Jul 27 21:30:16.408829 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.408756    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk" podUID="7acb5cf7-b965-4501-948b-49e3f93f155a" containerName="payload-processing" containerID="cri-o://945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00" gracePeriod=30
Jul 27 21:30:16.416060 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.416037    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"]
Jul 27 21:30:16.416253 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.416233    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d" podUID="ed6cec75-5242-43e8-b854-18100696d4de" containerName="payload-pre-processing" containerID="cri-o://28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea" gracePeriod=30
Jul 27 21:30:16.459215 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.459159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:16.527633 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.527574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:16.708974 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.708953    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:30:16.711733 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.711715    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:30:16.715239 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.715219    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:30:16.789117 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8fjj\" (UniqueName: \"kubernetes.io/projected/8767f7d1-44d0-4300-8d68-7994a5fc45b2-kube-api-access-v8fjj\") pod \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") "
Jul 27 21:30:16.789334 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789157    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls\") pod \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") "
Jul 27 21:30:16.789334 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6cec75-5242-43e8-b854-18100696d4de-plugins-config-volume\") pod \"ed6cec75-5242-43e8-b854-18100696d4de\" (UID: \"ed6cec75-5242-43e8-b854-18100696d4de\") "
Jul 27 21:30:16.789462 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789340    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acb5cf7-b965-4501-948b-49e3f93f155a-plugins-config-volume\") pod \"7acb5cf7-b965-4501-948b-49e3f93f155a\" (UID: \"7acb5cf7-b965-4501-948b-49e3f93f155a\") "
Jul 27 21:30:16.789520 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8767f7d1-44d0-4300-8d68-7994a5fc45b2-openshift-service-ca\") pod \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\" (UID: \"8767f7d1-44d0-4300-8d68-7994a5fc45b2\") "
Jul 27 21:30:16.789520 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlxwc\" (UniqueName: \"kubernetes.io/projected/7acb5cf7-b965-4501-948b-49e3f93f155a-kube-api-access-rlxwc\") pod \"7acb5cf7-b965-4501-948b-49e3f93f155a\" (UID: \"7acb5cf7-b965-4501-948b-49e3f93f155a\") "
Jul 27 21:30:16.789520 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789512    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9546s\" (UniqueName: \"kubernetes.io/projected/ed6cec75-5242-43e8-b854-18100696d4de-kube-api-access-9546s\") pod \"ed6cec75-5242-43e8-b854-18100696d4de\" (UID: \"ed6cec75-5242-43e8-b854-18100696d4de\") "
Jul 27 21:30:16.789686 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789517    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6cec75-5242-43e8-b854-18100696d4de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed6cec75-5242-43e8-b854-18100696d4de" (UID: "ed6cec75-5242-43e8-b854-18100696d4de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:30:16.789893 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789841    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7acb5cf7-b965-4501-948b-49e3f93f155a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7acb5cf7-b965-4501-948b-49e3f93f155a" (UID: "7acb5cf7-b965-4501-948b-49e3f93f155a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:30:16.789893 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8767f7d1-44d0-4300-8d68-7994a5fc45b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8767f7d1-44d0-4300-8d68-7994a5fc45b2" (UID: "8767f7d1-44d0-4300-8d68-7994a5fc45b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:30:16.790299 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789955    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6cec75-5242-43e8-b854-18100696d4de-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:30:16.790299 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789969    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7acb5cf7-b965-4501-948b-49e3f93f155a-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:30:16.790299 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.789979    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8767f7d1-44d0-4300-8d68-7994a5fc45b2-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:30:16.791491 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.791463    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8767f7d1-44d0-4300-8d68-7994a5fc45b2" (UID: "8767f7d1-44d0-4300-8d68-7994a5fc45b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:30:16.791585 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.791513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8767f7d1-44d0-4300-8d68-7994a5fc45b2-kube-api-access-v8fjj" (OuterVolumeSpecName: "kube-api-access-v8fjj") pod "8767f7d1-44d0-4300-8d68-7994a5fc45b2" (UID: "8767f7d1-44d0-4300-8d68-7994a5fc45b2"). InnerVolumeSpecName "kube-api-access-v8fjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:30:16.791745 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.791717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6cec75-5242-43e8-b854-18100696d4de-kube-api-access-9546s" (OuterVolumeSpecName: "kube-api-access-9546s") pod "ed6cec75-5242-43e8-b854-18100696d4de" (UID: "ed6cec75-5242-43e8-b854-18100696d4de"). InnerVolumeSpecName "kube-api-access-9546s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:30:16.791818 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.791753    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acb5cf7-b965-4501-948b-49e3f93f155a-kube-api-access-rlxwc" (OuterVolumeSpecName: "kube-api-access-rlxwc") pod "7acb5cf7-b965-4501-948b-49e3f93f155a" (UID: "7acb5cf7-b965-4501-948b-49e3f93f155a"). InnerVolumeSpecName "kube-api-access-rlxwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:30:16.891479 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.891386    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlxwc\" (UniqueName: \"kubernetes.io/projected/7acb5cf7-b965-4501-948b-49e3f93f155a-kube-api-access-rlxwc\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:30:16.891479 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.891421    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9546s\" (UniqueName: \"kubernetes.io/projected/ed6cec75-5242-43e8-b854-18100696d4de-kube-api-access-9546s\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:30:16.891479 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.891430    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8fjj\" (UniqueName: \"kubernetes.io/projected/8767f7d1-44d0-4300-8d68-7994a5fc45b2-kube-api-access-v8fjj\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:30:16.891479 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.891439    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8767f7d1-44d0-4300-8d68-7994a5fc45b2-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:30:16.991282 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.991244    2577 generic.go:356] "Generic (PLEG): container finished" podID="ed6cec75-5242-43e8-b854-18100696d4de" containerID="28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea" exitCode=0
Jul 27 21:30:16.991461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.991290    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"
Jul 27 21:30:16.991461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.991338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d" event={"ID":"ed6cec75-5242-43e8-b854-18100696d4de","Type":"ContainerDied","Data":"28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea"}
Jul 27 21:30:16.991461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.991397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d" event={"ID":"ed6cec75-5242-43e8-b854-18100696d4de","Type":"ContainerDied","Data":"20796fccf239a105ca23c69a17f062ac757a351faff4b0e43427eeef2c75f0d7"}
Jul 27 21:30:16.991461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.991422    2577 scope.go:117] "RemoveContainer" containerID="28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea"
Jul 27 21:30:16.992783 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.992762    2577 generic.go:356] "Generic (PLEG): container finished" podID="7acb5cf7-b965-4501-948b-49e3f93f155a" containerID="945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00" exitCode=0
Jul 27 21:30:16.992895 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.992814    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"
Jul 27 21:30:16.992895 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.992848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk" event={"ID":"7acb5cf7-b965-4501-948b-49e3f93f155a","Type":"ContainerDied","Data":"945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00"}
Jul 27 21:30:16.993007 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.992911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk" event={"ID":"7acb5cf7-b965-4501-948b-49e3f93f155a","Type":"ContainerDied","Data":"656ce03c20ff793548800bdb4d4bc118e058925758a1dfe8b9af13809d8e0e09"}
Jul 27 21:30:16.994270 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.994248    2577 generic.go:356] "Generic (PLEG): container finished" podID="8767f7d1-44d0-4300-8d68-7994a5fc45b2" containerID="3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef" exitCode=0
Jul 27 21:30:16.994361 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.994277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88" event={"ID":"8767f7d1-44d0-4300-8d68-7994a5fc45b2","Type":"ContainerDied","Data":"3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef"}
Jul 27 21:30:16.994361 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.994306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88" event={"ID":"8767f7d1-44d0-4300-8d68-7994a5fc45b2","Type":"ContainerDied","Data":"44656605c972b71e927ff4fa289c8289fa03b35ee487bf17433fcc444ee6f0f6"}
Jul 27 21:30:16.994361 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:16.994307    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"
Jul 27 21:30:17.002393 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.002346    2577 scope.go:117] "RemoveContainer" containerID="28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea"
Jul 27 21:30:17.002668 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:30:17.002646    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea\": container with ID starting with 28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea not found: ID does not exist" containerID="28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea"
Jul 27 21:30:17.002783 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.002678    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea"} err="failed to get container status \"28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea\": rpc error: code = NotFound desc = could not find container \"28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea\": container with ID starting with 28ac792e0f33f28b0aa919a2e0fa2cf0843d1e915a38661497750526de45daea not found: ID does not exist"
Jul 27 21:30:17.002783 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.002701    2577 scope.go:117] "RemoveContainer" containerID="945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00"
Jul 27 21:30:17.011081 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.011065    2577 scope.go:117] "RemoveContainer" containerID="945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00"
Jul 27 21:30:17.011403 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:30:17.011384    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00\": container with ID starting with 945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00 not found: ID does not exist" containerID="945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00"
Jul 27 21:30:17.011469 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.011410    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00"} err="failed to get container status \"945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00\": rpc error: code = NotFound desc = could not find container \"945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00\": container with ID starting with 945f0f899a7b3dc910ab55b1ecae333c8a36d174534d3b262d5844dcefeefc00 not found: ID does not exist"
Jul 27 21:30:17.011469 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.011427    2577 scope.go:117] "RemoveContainer" containerID="3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef"
Jul 27 21:30:17.018899 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.018882    2577 scope.go:117] "RemoveContainer" containerID="3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef"
Jul 27 21:30:17.019146 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:30:17.019126    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef\": container with ID starting with 3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef not found: ID does not exist" containerID="3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef"
Jul 27 21:30:17.019230 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.019161    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef"} err="failed to get container status \"3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef\": rpc error: code = NotFound desc = could not find container \"3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef\": container with ID starting with 3565503a9507ad0e284857d064bdd464c46c495850517b60c0655d22de2263ef not found: ID does not exist"
Jul 27 21:30:17.019230 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.019133    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"]
Jul 27 21:30:17.023240 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.023219    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86cc77-868cd448cb-hjd4d"]
Jul 27 21:30:17.034245 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.034223    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"]
Jul 27 21:30:17.036457 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.036437    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-86cc77-647fd6c6f-bzq88"]
Jul 27 21:30:17.049263 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.049246    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"]
Jul 27 21:30:17.051220 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.051170    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86cc77-b5dd5f7d-g7sfk"]
Jul 27 21:30:17.185256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.185144    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7acb5cf7-b965-4501-948b-49e3f93f155a" path="/var/lib/kubelet/pods/7acb5cf7-b965-4501-948b-49e3f93f155a/volumes"
Jul 27 21:30:17.185627 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.185608    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8767f7d1-44d0-4300-8d68-7994a5fc45b2" path="/var/lib/kubelet/pods/8767f7d1-44d0-4300-8d68-7994a5fc45b2/volumes"
Jul 27 21:30:17.185907 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.185890    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d" path="/var/lib/kubelet/pods/ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d/volumes"
Jul 27 21:30:17.186234 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:17.186218    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed6cec75-5242-43e8-b854-18100696d4de" path="/var/lib/kubelet/pods/ed6cec75-5242-43e8-b854-18100696d4de/volumes"
Jul 27 21:30:20.454909 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:20.454866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:20.838952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:20.838909    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:21.154956 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:21.154857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:30:26.793698 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:26.793653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:26.914477 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:26.914439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:27.135837 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:27.135747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:27.356304 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:27.356273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:27.690908 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:27.690861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:28.010121 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:28.010066    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:28.150125 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:28.150090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:31.516886 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.516843    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:31.609716 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.609653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:31.614004 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.613981    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"]
Jul 27 21:30:31.614446 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614428    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d" containerName="cleanup"
Jul 27 21:30:31.614501 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614449    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d" containerName="cleanup"
Jul 27 21:30:31.614501 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614480    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7acb5cf7-b965-4501-948b-49e3f93f155a" containerName="payload-processing"
Jul 27 21:30:31.614501 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614488    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7acb5cf7-b965-4501-948b-49e3f93f155a" containerName="payload-processing"
Jul 27 21:30:31.614600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614505    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae0274c-74c9-43ec-bedb-8915a04ee87f" containerName="cleanup"
Jul 27 21:30:31.614600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614514    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae0274c-74c9-43ec-bedb-8915a04ee87f" containerName="cleanup"
Jul 27 21:30:31.614600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614528    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6cec75-5242-43e8-b854-18100696d4de" containerName="payload-pre-processing"
Jul 27 21:30:31.614600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614535    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6cec75-5242-43e8-b854-18100696d4de" containerName="payload-pre-processing"
Jul 27 21:30:31.614600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614548    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8767f7d1-44d0-4300-8d68-7994a5fc45b2" containerName="maas-api"
Jul 27 21:30:31.614600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614555    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8767f7d1-44d0-4300-8d68-7994a5fc45b2" containerName="maas-api"
Jul 27 21:30:31.614794 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614627    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8767f7d1-44d0-4300-8d68-7994a5fc45b2" containerName="maas-api"
Jul 27 21:30:31.614794 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614639    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7acb5cf7-b965-4501-948b-49e3f93f155a" containerName="payload-processing"
Jul 27 21:30:31.614794 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614650    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae0274c-74c9-43ec-bedb-8915a04ee87f" containerName="cleanup"
Jul 27 21:30:31.614794 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614661    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6cec75-5242-43e8-b854-18100696d4de" containerName="payload-pre-processing"
Jul 27 21:30:31.614794 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.614669    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae1ab76f-9286-4e76-88a5-eb75c6ae4b5d" containerName="cleanup"
Jul 27 21:30:31.617901 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.617886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:31.620771 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.620750    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 21:30:31.621302 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.621282    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:30:31.621981 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.621966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-c910ce-serving-cert\""
Jul 27 21:30:31.622067 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.621970    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cgjq6\""
Jul 27 21:30:31.629464 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.629440    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"]
Jul 27 21:30:31.696936 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.696898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:31.724999 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.724967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54d20a9-a111-4653-bbb9-ea5cab26bf77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:31.725157 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.725058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8dj\" (UniqueName: \"kubernetes.io/projected/d54d20a9-a111-4653-bbb9-ea5cab26bf77-kube-api-access-hq8dj\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:31.725157 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.725081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:31.826621 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.826517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8dj\" (UniqueName: \"kubernetes.io/projected/d54d20a9-a111-4653-bbb9-ea5cab26bf77-kube-api-access-hq8dj\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:31.826621 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.826569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:31.826849 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.826630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54d20a9-a111-4653-bbb9-ea5cab26bf77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:31.826849 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:30:31.826734    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-serving-cert: secret "maas-api-e2e-shared-a-c910ce-serving-cert" not found
Jul 27 21:30:31.826849 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:30:31.826806    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls podName:d54d20a9-a111-4653-bbb9-ea5cab26bf77 nodeName:}" failed. No retries permitted until 2026-07-27 21:30:32.326786032 +0000 UTC m=+2349.692351877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls") pod "maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2" (UID: "d54d20a9-a111-4653-bbb9-ea5cab26bf77") : secret "maas-api-e2e-shared-a-c910ce-serving-cert" not found
Jul 27 21:30:31.827386 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.827369    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54d20a9-a111-4653-bbb9-ea5cab26bf77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:31.837818 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:31.837794    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8dj\" (UniqueName: \"kubernetes.io/projected/d54d20a9-a111-4653-bbb9-ea5cab26bf77-kube-api-access-hq8dj\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:32.214346 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.214253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:32.312394 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.312354    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:32.331529 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.331498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:32.334015 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.333993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls\") pod \"maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:32.365077 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.365047    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"]
Jul 27 21:30:32.373728 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.373701    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:32.381261 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.381102    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-c910ce\""
Jul 27 21:30:32.381399 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.381325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-c910ce-dockercfg-5t8w5\""
Jul 27 21:30:32.383001 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.382980    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"]
Jul 27 21:30:32.414514 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.414484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:32.419164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.419144    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"]
Jul 27 21:30:32.422524 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.422508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:32.432095 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.432072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8ca59b-b54e-43b4-84d0-632257bf9118-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7\" (UID: \"4c8ca59b-b54e-43b4-84d0-632257bf9118\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:32.432226 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.432112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5pxv\" (UniqueName: \"kubernetes.io/projected/4c8ca59b-b54e-43b4-84d0-632257bf9118-kube-api-access-q5pxv\") pod \"payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7\" (UID: \"4c8ca59b-b54e-43b4-84d0-632257bf9118\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:32.434810 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.434790    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"]
Jul 27 21:30:32.530208 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.530165    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:32.532984 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.532960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8ca59b-b54e-43b4-84d0-632257bf9118-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7\" (UID: \"4c8ca59b-b54e-43b4-84d0-632257bf9118\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:32.533102 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.532998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f559ef-6971-489d-9076-c5a19212336f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt\" (UID: \"f9f559ef-6971-489d-9076-c5a19212336f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:32.533102 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.533033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5pxv\" (UniqueName: \"kubernetes.io/projected/4c8ca59b-b54e-43b4-84d0-632257bf9118-kube-api-access-q5pxv\") pod \"payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7\" (UID: \"4c8ca59b-b54e-43b4-84d0-632257bf9118\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:32.533524 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.533158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz6jw\" (UniqueName: \"kubernetes.io/projected/f9f559ef-6971-489d-9076-c5a19212336f-kube-api-access-vz6jw\") pod \"payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt\" (UID: \"f9f559ef-6971-489d-9076-c5a19212336f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:32.533643 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.533627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8ca59b-b54e-43b4-84d0-632257bf9118-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7\" (UID: \"4c8ca59b-b54e-43b4-84d0-632257bf9118\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:32.543037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.543013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5pxv\" (UniqueName: \"kubernetes.io/projected/4c8ca59b-b54e-43b4-84d0-632257bf9118-kube-api-access-q5pxv\") pod \"payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7\" (UID: \"4c8ca59b-b54e-43b4-84d0-632257bf9118\") " pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:32.633801 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.633764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz6jw\" (UniqueName: \"kubernetes.io/projected/f9f559ef-6971-489d-9076-c5a19212336f-kube-api-access-vz6jw\") pod \"payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt\" (UID: \"f9f559ef-6971-489d-9076-c5a19212336f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:32.634003 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.633838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f559ef-6971-489d-9076-c5a19212336f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt\" (UID: \"f9f559ef-6971-489d-9076-c5a19212336f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:32.634624 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.634600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f559ef-6971-489d-9076-c5a19212336f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt\" (UID: \"f9f559ef-6971-489d-9076-c5a19212336f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:32.642424 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.642401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz6jw\" (UniqueName: \"kubernetes.io/projected/f9f559ef-6971-489d-9076-c5a19212336f-kube-api-access-vz6jw\") pod \"payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt\" (UID: \"f9f559ef-6971-489d-9076-c5a19212336f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:32.668647 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.668625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"]
Jul 27 21:30:32.670510 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:30:32.670484    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54d20a9_a111_4653_bbb9_ea5cab26bf77.slice/crio-c042e5033c93c867fd303e906faf57dc46684e034a452bb11712f1b683279638 WatchSource:0}: Error finding container c042e5033c93c867fd303e906faf57dc46684e034a452bb11712f1b683279638: Status 404 returned error can't find the container with id c042e5033c93c867fd303e906faf57dc46684e034a452bb11712f1b683279638
Jul 27 21:30:32.685322 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.685303    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:32.732479 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.732444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:32.829166 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.828998    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"]
Jul 27 21:30:32.831373 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:30:32.831335    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8ca59b_b54e_43b4_84d0_632257bf9118.slice/crio-32228490b84da021db75578bf1a77a2b2da5e195d5576086bdd1ab4fada53a66 WatchSource:0}: Error finding container 32228490b84da021db75578bf1a77a2b2da5e195d5576086bdd1ab4fada53a66: Status 404 returned error can't find the container with id 32228490b84da021db75578bf1a77a2b2da5e195d5576086bdd1ab4fada53a66
Jul 27 21:30:32.877624 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:32.877556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"]
Jul 27 21:30:32.879123 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:30:32.879098    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f559ef_6971_489d_9076_c5a19212336f.slice/crio-2299fd156649a818989c60fe8246640e8f4dc90451f7b42465caff72dbd99106 WatchSource:0}: Error finding container 2299fd156649a818989c60fe8246640e8f4dc90451f7b42465caff72dbd99106: Status 404 returned error can't find the container with id 2299fd156649a818989c60fe8246640e8f4dc90451f7b42465caff72dbd99106
Jul 27 21:30:33.080413 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.080319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7" event={"ID":"4c8ca59b-b54e-43b4-84d0-632257bf9118","Type":"ContainerStarted","Data":"4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77"}
Jul 27 21:30:33.080413 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.080378    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7" event={"ID":"4c8ca59b-b54e-43b4-84d0-632257bf9118","Type":"ContainerStarted","Data":"32228490b84da021db75578bf1a77a2b2da5e195d5576086bdd1ab4fada53a66"}
Jul 27 21:30:33.080766 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.080500    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:33.081559 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.081536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2" event={"ID":"d54d20a9-a111-4653-bbb9-ea5cab26bf77","Type":"ContainerStarted","Data":"c042e5033c93c867fd303e906faf57dc46684e034a452bb11712f1b683279638"}
Jul 27 21:30:33.083110 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.083087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt" event={"ID":"f9f559ef-6971-489d-9076-c5a19212336f","Type":"ContainerStarted","Data":"68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d"}
Jul 27 21:30:33.083244 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.083129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt" event={"ID":"f9f559ef-6971-489d-9076-c5a19212336f","Type":"ContainerStarted","Data":"2299fd156649a818989c60fe8246640e8f4dc90451f7b42465caff72dbd99106"}
Jul 27 21:30:33.083244 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.083167    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:33.097700 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.097659    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7" podStartSLOduration=1.097646501 podStartE2EDuration="1.097646501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:33.096892881 +0000 UTC m=+2350.462458728" watchObservedRunningTime="2026-07-27 21:30:33.097646501 +0000 UTC m=+2350.463212326"
Jul 27 21:30:33.116783 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:33.116740    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt" podStartSLOduration=1.11672728 podStartE2EDuration="1.11672728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:33.115807955 +0000 UTC m=+2350.481373812" watchObservedRunningTime="2026-07-27 21:30:33.11672728 +0000 UTC m=+2350.482293129"
Jul 27 21:30:34.091025 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:34.090983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2" event={"ID":"d54d20a9-a111-4653-bbb9-ea5cab26bf77","Type":"ContainerStarted","Data":"b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25"}
Jul 27 21:30:34.109982 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:34.109930    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2" podStartSLOduration=2.430577683 podStartE2EDuration="3.109915086s" totalImagesPullingTime="679.337403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:34.107891314 +0000 UTC m=+2351.473457161" watchObservedRunningTime="2026-07-27 21:30:34.109915086 +0000 UTC m=+2351.475480933"
Jul 27 21:30:35.095723 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:35.095687    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:35.928888 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:35.928850    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:36.258538 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:36.258501    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:36.572617 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:36.572527    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:36.694761 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:36.694723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:39.155105 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.155066    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:30:39.479649 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.479615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:39.542958 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.542923    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"]
Jul 27 21:30:39.546693 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.546674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.549412 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.549372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-85f004-serving-cert\""
Jul 27 21:30:39.559164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.559139    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"]
Jul 27 21:30:39.586515 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.586478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:39.673937 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.673900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:39.698423 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.698383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.698599 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.698455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjx5\" (UniqueName: \"kubernetes.io/projected/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-kube-api-access-ztjx5\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.698599 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.698504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.769974 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.769893    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:39.799779 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.799748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.799963 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.799833    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjx5\" (UniqueName: \"kubernetes.io/projected/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-kube-api-access-ztjx5\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.799963 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.799878    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.800380 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.800360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.802376 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.802354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.808542 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.808521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjx5\" (UniqueName: \"kubernetes.io/projected/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-kube-api-access-ztjx5\") pod \"maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.860070 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.860016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:39.999806 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:39.999777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"]
Jul 27 21:30:40.001812 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:30:40.001788    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed43bbb_ea5d_486c_bb0c_1d7ae8b9e1d6.slice/crio-210dfd843435f5cf3bfe5f60d59a24012bd2f0f2a361200b8c6383e8164a4d80 WatchSource:0}: Error finding container 210dfd843435f5cf3bfe5f60d59a24012bd2f0f2a361200b8c6383e8164a4d80: Status 404 returned error can't find the container with id 210dfd843435f5cf3bfe5f60d59a24012bd2f0f2a361200b8c6383e8164a4d80
Jul 27 21:30:40.126866 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.126778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj" event={"ID":"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6","Type":"ContainerStarted","Data":"210dfd843435f5cf3bfe5f60d59a24012bd2f0f2a361200b8c6383e8164a4d80"}
Jul 27 21:30:40.178960 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.178890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:40.282979 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.282945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:40.381086 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.381027    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"]
Jul 27 21:30:40.384455 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.384439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:40.388644 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.388606    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-85f004\""
Jul 27 21:30:40.388868 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.388626    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-85f004-dockercfg-f8h4v\""
Jul 27 21:30:40.407344 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.407316    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"]
Jul 27 21:30:40.462612 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.462582    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"]
Jul 27 21:30:40.466157 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.466134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:40.483106 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.483080    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"]
Jul 27 21:30:40.507892 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.507854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f077dd-924a-4915-8ae0-64a0ec409001-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7\" (UID: \"a1f077dd-924a-4915-8ae0-64a0ec409001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:40.508063 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.507904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phk95\" (UniqueName: \"kubernetes.io/projected/a1f077dd-924a-4915-8ae0-64a0ec409001-kube-api-access-phk95\") pod \"payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7\" (UID: \"a1f077dd-924a-4915-8ae0-64a0ec409001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:40.608386 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.608354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bjn\" (UniqueName: \"kubernetes.io/projected/14b42c47-4336-4d19-a337-71bd80d68c67-kube-api-access-75bjn\") pod \"payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6\" (UID: \"14b42c47-4336-4d19-a337-71bd80d68c67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:40.608563 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.608399    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b42c47-4336-4d19-a337-71bd80d68c67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6\" (UID: \"14b42c47-4336-4d19-a337-71bd80d68c67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:40.608563 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.608477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f077dd-924a-4915-8ae0-64a0ec409001-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7\" (UID: \"a1f077dd-924a-4915-8ae0-64a0ec409001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:40.608563 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.608531    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phk95\" (UniqueName: \"kubernetes.io/projected/a1f077dd-924a-4915-8ae0-64a0ec409001-kube-api-access-phk95\") pod \"payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7\" (UID: \"a1f077dd-924a-4915-8ae0-64a0ec409001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:40.609495 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.609068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f077dd-924a-4915-8ae0-64a0ec409001-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7\" (UID: \"a1f077dd-924a-4915-8ae0-64a0ec409001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:40.621652 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.621622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phk95\" (UniqueName: \"kubernetes.io/projected/a1f077dd-924a-4915-8ae0-64a0ec409001-kube-api-access-phk95\") pod \"payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7\" (UID: \"a1f077dd-924a-4915-8ae0-64a0ec409001\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:40.694763 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.694680    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:40.709285 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.709250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bjn\" (UniqueName: \"kubernetes.io/projected/14b42c47-4336-4d19-a337-71bd80d68c67-kube-api-access-75bjn\") pod \"payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6\" (UID: \"14b42c47-4336-4d19-a337-71bd80d68c67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:40.709406 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.709313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b42c47-4336-4d19-a337-71bd80d68c67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6\" (UID: \"14b42c47-4336-4d19-a337-71bd80d68c67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:40.710031 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.710009    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b42c47-4336-4d19-a337-71bd80d68c67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6\" (UID: \"14b42c47-4336-4d19-a337-71bd80d68c67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:40.718492 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.718468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bjn\" (UniqueName: \"kubernetes.io/projected/14b42c47-4336-4d19-a337-71bd80d68c67-kube-api-access-75bjn\") pod \"payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6\" (UID: \"14b42c47-4336-4d19-a337-71bd80d68c67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:40.776370 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.776338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:40.839140 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.838918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"]
Jul 27 21:30:40.842790 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:30:40.842671    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f077dd_924a_4915_8ae0_64a0ec409001.slice/crio-9e878d76c5576832a07e2bb07681c359f8b4e8b66ab72c75c7287dfd4f812615 WatchSource:0}: Error finding container 9e878d76c5576832a07e2bb07681c359f8b4e8b66ab72c75c7287dfd4f812615: Status 404 returned error can't find the container with id 9e878d76c5576832a07e2bb07681c359f8b4e8b66ab72c75c7287dfd4f812615
Jul 27 21:30:40.922847 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:40.922769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"]
Jul 27 21:30:40.925564 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:30:40.925538    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b42c47_4336_4d19_a337_71bd80d68c67.slice/crio-21f55a6ef3f0618d1075ad037b331272020e1b0530a37548880b35c03cea4a85 WatchSource:0}: Error finding container 21f55a6ef3f0618d1075ad037b331272020e1b0530a37548880b35c03cea4a85: Status 404 returned error can't find the container with id 21f55a6ef3f0618d1075ad037b331272020e1b0530a37548880b35c03cea4a85
Jul 27 21:30:41.106658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.106627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:30:41.134490 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.134447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj" event={"ID":"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6","Type":"ContainerStarted","Data":"042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf"}
Jul 27 21:30:41.134693 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.134661    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:41.136207 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.136156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6" event={"ID":"14b42c47-4336-4d19-a337-71bd80d68c67","Type":"ContainerStarted","Data":"8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd"}
Jul 27 21:30:41.136355 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.136227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6" event={"ID":"14b42c47-4336-4d19-a337-71bd80d68c67","Type":"ContainerStarted","Data":"21f55a6ef3f0618d1075ad037b331272020e1b0530a37548880b35c03cea4a85"}
Jul 27 21:30:41.136355 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.136255    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:41.138123 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.138097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7" event={"ID":"a1f077dd-924a-4915-8ae0-64a0ec409001","Type":"ContainerStarted","Data":"0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576"}
Jul 27 21:30:41.138257 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.138148    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:30:41.138257 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.138172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7" event={"ID":"a1f077dd-924a-4915-8ae0-64a0ec409001","Type":"ContainerStarted","Data":"9e878d76c5576832a07e2bb07681c359f8b4e8b66ab72c75c7287dfd4f812615"}
Jul 27 21:30:41.153417 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.153368    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj" podStartSLOduration=1.834847452 podStartE2EDuration="2.153353461s" totalImagesPullingTime="318.506009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:41.152264579 +0000 UTC m=+2358.517830427" watchObservedRunningTime="2026-07-27 21:30:41.153353461 +0000 UTC m=+2358.518919286"
Jul 27 21:30:41.170408 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.170360    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6" podStartSLOduration=1.170345453 podStartE2EDuration="1.170345453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:41.169619507 +0000 UTC m=+2358.535185352" watchObservedRunningTime="2026-07-27 21:30:41.170345453 +0000 UTC m=+2358.535911299"
Jul 27 21:30:41.185666 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:41.185620    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7" podStartSLOduration=1.1856062330000001 podStartE2EDuration="1.185606233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:41.184790822 +0000 UTC m=+2358.550356670" watchObservedRunningTime="2026-07-27 21:30:41.185606233 +0000 UTC m=+2358.551172079"
Jul 27 21:30:43.954067 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:43.954031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:30:44.093389 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:44.093358    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:30:44.093692 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:44.093675    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:30:47.148106 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:47.148080    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:30:51.159159 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:51.159123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:30:52.144584 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:52.144556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:30:52.144761 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:30:52.144633    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:31:03.731869 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.731833    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"]
Jul 27 21:31:03.735517 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.735495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.738305 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.738288    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c910ce\"/\"default-dockercfg-k6k7w\""
Jul 27 21:31:03.739434 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.739411    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c910ce\"/\"openshift-service-ca.crt\""
Jul 27 21:31:03.739434 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.739425    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c910ce\"/\"kube-root-ca.crt\""
Jul 27 21:31:03.739756 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.739735    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c910ce\"/\"auth-test-model-c910ce-kserve-self-signed-certs\""
Jul 27 21:31:03.759790 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.759759    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"]
Jul 27 21:31:03.781996 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.781952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:03.818199 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.818146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tmp-dir\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.818360 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.818251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-model-cache\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.818360 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.818300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smvt\" (UniqueName: \"kubernetes.io/projected/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-kube-api-access-4smvt\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.818360 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.818319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-home\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.818482 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.818373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-dshm\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.818482 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.818409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tls-certs\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.919866 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.919830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smvt\" (UniqueName: \"kubernetes.io/projected/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-kube-api-access-4smvt\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.919866 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.919869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-home\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.920075 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.919895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-dshm\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.920075 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.919924    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tls-certs\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.920075 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.919974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tmp-dir\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.920229 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.920113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-model-cache\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.920418 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.920386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-home\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.920503 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.920416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tmp-dir\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.920503 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.920474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-model-cache\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.922403 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.922385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-dshm\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.922572 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.922558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tls-certs\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:03.929695 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:03.929670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smvt\" (UniqueName: \"kubernetes.io/projected/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-kube-api-access-4smvt\") pod \"auth-test-model-c910ce-kserve-696776554c-7lp5c\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") " pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:04.046491 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:04.046394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:04.182346 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:04.182314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"]
Jul 27 21:31:04.185602 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:31:04.185571    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41642d11_5b0b_46ce_b2fa_8563cf7b27cd.slice/crio-5116f519f1702c350335365cd1e6139931b8f76f3ee75efd2c43475493c95048 WatchSource:0}: Error finding container 5116f519f1702c350335365cd1e6139931b8f76f3ee75efd2c43475493c95048: Status 404 returned error can't find the container with id 5116f519f1702c350335365cd1e6139931b8f76f3ee75efd2c43475493c95048
Jul 27 21:31:04.261533 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:04.261499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c" event={"ID":"41642d11-5b0b-46ce-b2fa-8563cf7b27cd","Type":"ContainerStarted","Data":"148f39e24a2a0324ddb3ff6cfce81e581d8c6a5c3b41cb7e6b83c906c8af94c2"}
Jul 27 21:31:04.261533 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:04.261541    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c" event={"ID":"41642d11-5b0b-46ce-b2fa-8563cf7b27cd","Type":"ContainerStarted","Data":"5116f519f1702c350335365cd1e6139931b8f76f3ee75efd2c43475493c95048"}
Jul 27 21:31:04.281703 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:04.281653    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c" podStartSLOduration=1.281636673 podStartE2EDuration="1.281636673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:04.281373071 +0000 UTC m=+2381.646938918" watchObservedRunningTime="2026-07-27 21:31:04.281636673 +0000 UTC m=+2381.647202520"
Jul 27 21:31:07.252587 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:07.252546    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:31:14.047470 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:14.047427    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:14.047470 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:14.047470    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:14.059961 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:14.059934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:14.325842 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:14.325763    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:31:28.088112 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:28.088070    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:31.456689 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:31.456655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:31:53.100291 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:53.100247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:31:55.554977 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:55.554946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:31:58.013675 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:31:58.013636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:32:16.160928 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:32:16.160895    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:32:29.258975 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:32:29.258940    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:32:41.865004 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:32:41.864973    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:32:59.761994 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:32:59.761956    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:33:10.023707 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:33:10.023664    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:33:14.450725 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:33:14.450644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:33:15.818728 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:33:15.818697    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:33:15.871964 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:33:15.871927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:33:34.559857 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:33:34.559822    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:33:53.457330 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:33:53.457298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:34:14.759483 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:34:14.759446    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:34:29.150723 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:34:29.150686    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:34:41.265656 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:34:41.265621    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:34:56.852930 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:34:56.852895    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:35:14.455742 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:14.455705    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:35:23.480691 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:23.480661    2577 scope.go:117] "RemoveContainer" containerID="86a6c3ce76c3eee7da324523f761d5ca79aba98d8b1a7875f100ae1418c132d4"
Jul 27 21:35:35.063595 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:35.063558    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:35:39.539735 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:39.539694    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"]
Jul 27 21:35:39.592073 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:39.592013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:35:41.092084 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:41.092039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:35:51.257789 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.257751    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:35:51.650903 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.650831    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"]
Jul 27 21:35:51.654529 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.654514    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.658268 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.658239    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c910ce\"/\"sub-test-model-c910ce-kserve-self-signed-certs\""
Jul 27 21:35:51.672088 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.672062    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"]
Jul 27 21:35:51.681468 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.681441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:35:51.813801 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.813762    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-model-cache\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.813801 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.813803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626fc1d4-85c1-4005-a921-73ab4f1248ee-tls-certs\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.814048 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.813876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-dshm\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.814048 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.813916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-tmp-dir\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.814048 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.813937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwsv\" (UniqueName: \"kubernetes.io/projected/626fc1d4-85c1-4005-a921-73ab4f1248ee-kube-api-access-phwsv\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.814048 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.813957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-home\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.914868 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.914784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-home\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.915020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.914870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-model-cache\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.915020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.914894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626fc1d4-85c1-4005-a921-73ab4f1248ee-tls-certs\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.915020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.914963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-dshm\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.915020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.914986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-tmp-dir\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.915020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.915009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwsv\" (UniqueName: \"kubernetes.io/projected/626fc1d4-85c1-4005-a921-73ab4f1248ee-kube-api-access-phwsv\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.915321 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.915303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-home\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.915445 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.915348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-model-cache\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.915445 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.915368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-tmp-dir\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.917267 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.917247    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-dshm\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.917620 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.917601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626fc1d4-85c1-4005-a921-73ab4f1248ee-tls-certs\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.926952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.926932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwsv\" (UniqueName: \"kubernetes.io/projected/626fc1d4-85c1-4005-a921-73ab4f1248ee-kube-api-access-phwsv\") pod \"sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") " pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:51.964000 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:51.963961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:35:52.102738 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:52.102708    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"]
Jul 27 21:35:52.105073 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:35:52.105050    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod626fc1d4_85c1_4005_a921_73ab4f1248ee.slice/crio-6b2af6be405726c285563c44bc17c53ac1d185a761b911855cc25d6988ef7ca2 WatchSource:0}: Error finding container 6b2af6be405726c285563c44bc17c53ac1d185a761b911855cc25d6988ef7ca2: Status 404 returned error can't find the container with id 6b2af6be405726c285563c44bc17c53ac1d185a761b911855cc25d6988ef7ca2
Jul 27 21:35:52.688730 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:52.688684    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7" event={"ID":"626fc1d4-85c1-4005-a921-73ab4f1248ee","Type":"ContainerStarted","Data":"8528601d552e5238e243185e2c5876b68a7673b20f86fc16ec844d7db8bb41ea"}
Jul 27 21:35:52.688730 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:52.688728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7" event={"ID":"626fc1d4-85c1-4005-a921-73ab4f1248ee","Type":"ContainerStarted","Data":"6b2af6be405726c285563c44bc17c53ac1d185a761b911855cc25d6988ef7ca2"}
Jul 27 21:35:52.716675 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:52.716630    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7" podStartSLOduration=1.716616811 podStartE2EDuration="1.716616811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:35:52.715678216 +0000 UTC m=+2670.081244064" watchObservedRunningTime="2026-07-27 21:35:52.716616811 +0000 UTC m=+2670.082182658"
Jul 27 21:35:54.559105 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.559061    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c" podUID="41642d11-5b0b-46ce-b2fa-8563cf7b27cd" containerName="main" containerID="cri-o://148f39e24a2a0324ddb3ff6cfce81e581d8c6a5c3b41cb7e6b83c906c8af94c2" gracePeriod=45
Jul 27 21:35:54.704132 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.704094    2577 generic.go:356] "Generic (PLEG): container finished" podID="41642d11-5b0b-46ce-b2fa-8563cf7b27cd" containerID="148f39e24a2a0324ddb3ff6cfce81e581d8c6a5c3b41cb7e6b83c906c8af94c2" exitCode=0
Jul 27 21:35:54.704354 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.704172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c" event={"ID":"41642d11-5b0b-46ce-b2fa-8563cf7b27cd","Type":"ContainerDied","Data":"148f39e24a2a0324ddb3ff6cfce81e581d8c6a5c3b41cb7e6b83c906c8af94c2"}
Jul 27 21:35:54.821585 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.821511    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:35:54.841603 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.841570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-home\") pod \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") "
Jul 27 21:35:54.841786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.841612    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-model-cache\") pod \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") "
Jul 27 21:35:54.841786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.841637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-dshm\") pod \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") "
Jul 27 21:35:54.841786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.841684    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tls-certs\") pod \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") "
Jul 27 21:35:54.841786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.841719    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tmp-dir\") pod \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") "
Jul 27 21:35:54.841786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.841769    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4smvt\" (UniqueName: \"kubernetes.io/projected/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-kube-api-access-4smvt\") pod \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\" (UID: \"41642d11-5b0b-46ce-b2fa-8563cf7b27cd\") "
Jul 27 21:35:54.842046 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.841839    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "41642d11-5b0b-46ce-b2fa-8563cf7b27cd" (UID: "41642d11-5b0b-46ce-b2fa-8563cf7b27cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:35:54.842046 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.841855    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-home" (OuterVolumeSpecName: "home") pod "41642d11-5b0b-46ce-b2fa-8563cf7b27cd" (UID: "41642d11-5b0b-46ce-b2fa-8563cf7b27cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:35:54.842161 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.842049    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41642d11-5b0b-46ce-b2fa-8563cf7b27cd" (UID: "41642d11-5b0b-46ce-b2fa-8563cf7b27cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:35:54.842161 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.842120    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-home\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:35:54.842161 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.842133    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-model-cache\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:35:54.844029 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.844001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-dshm" (OuterVolumeSpecName: "dshm") pod "41642d11-5b0b-46ce-b2fa-8563cf7b27cd" (UID: "41642d11-5b0b-46ce-b2fa-8563cf7b27cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:35:54.844149 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.844083    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41642d11-5b0b-46ce-b2fa-8563cf7b27cd" (UID: "41642d11-5b0b-46ce-b2fa-8563cf7b27cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:35:54.844149 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.844112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-kube-api-access-4smvt" (OuterVolumeSpecName: "kube-api-access-4smvt") pod "41642d11-5b0b-46ce-b2fa-8563cf7b27cd" (UID: "41642d11-5b0b-46ce-b2fa-8563cf7b27cd"). InnerVolumeSpecName "kube-api-access-4smvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:35:54.943232 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.943175    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tls-certs\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:35:54.943232 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.943226    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-tmp-dir\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:35:54.943232 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.943236    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4smvt\" (UniqueName: \"kubernetes.io/projected/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-kube-api-access-4smvt\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:35:54.943489 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:54.943245    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41642d11-5b0b-46ce-b2fa-8563cf7b27cd-dshm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:35:55.710584 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:55.710535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c" event={"ID":"41642d11-5b0b-46ce-b2fa-8563cf7b27cd","Type":"ContainerDied","Data":"5116f519f1702c350335365cd1e6139931b8f76f3ee75efd2c43475493c95048"}
Jul 27 21:35:55.711177 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:55.710600    2577 scope.go:117] "RemoveContainer" containerID="148f39e24a2a0324ddb3ff6cfce81e581d8c6a5c3b41cb7e6b83c906c8af94c2"
Jul 27 21:35:55.711177 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:55.710551    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"
Jul 27 21:35:55.737355 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:55.737328    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"]
Jul 27 21:35:55.739069 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:55.739050    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/auth-test-model-c910ce-kserve-696776554c-7lp5c"]
Jul 27 21:35:57.186256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:35:57.186222    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41642d11-5b0b-46ce-b2fa-8563cf7b27cd" path="/var/lib/kubelet/pods/41642d11-5b0b-46ce-b2fa-8563cf7b27cd/volumes"
Jul 27 21:36:00.977438 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:00.977400    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:36:01.964931 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:01.964887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:36:01.965114 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:01.964950    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:36:01.977556 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:01.977530    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:36:02.755973 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:02.755943    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:36:18.325748 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.325670    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"]
Jul 27 21:36:18.326245 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.326230    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41642d11-5b0b-46ce-b2fa-8563cf7b27cd" containerName="main"
Jul 27 21:36:18.326245 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.326247    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41642d11-5b0b-46ce-b2fa-8563cf7b27cd" containerName="main"
Jul 27 21:36:18.326377 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.326331    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41642d11-5b0b-46ce-b2fa-8563cf7b27cd" containerName="main"
Jul 27 21:36:18.329588 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.329569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.333227 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.333205    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-85f004\"/\"openshift-service-ca.crt\""
Jul 27 21:36:18.333334 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.333179    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-85f004\"/\"kube-root-ca.crt\""
Jul 27 21:36:18.333334 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.333238    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-85f004\"/\"sub-test-model-85f004-kserve-self-signed-certs\""
Jul 27 21:36:18.333334 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.333260    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-85f004\"/\"default-dockercfg-59fsx\""
Jul 27 21:36:18.339446 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.339425    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"]
Jul 27 21:36:18.356627 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.356602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:36:18.456260 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.456225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-dshm\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.456459 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.456272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-home\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.456459 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.456334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvxz\" (UniqueName: \"kubernetes.io/projected/9ff91499-f7bc-439f-954b-e38d933f816a-kube-api-access-tdvxz\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.456459 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.456433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-model-cache\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.456627 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.456475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff91499-f7bc-439f-954b-e38d933f816a-tls-certs\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.456627 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.456515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-tmp-dir\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.557993 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.557958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvxz\" (UniqueName: \"kubernetes.io/projected/9ff91499-f7bc-439f-954b-e38d933f816a-kube-api-access-tdvxz\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.558222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.558019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-model-cache\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.558222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.558047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff91499-f7bc-439f-954b-e38d933f816a-tls-certs\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.558222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.558068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-tmp-dir\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.558222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.558109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-dshm\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.558222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.558140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-home\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.558556 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.558534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-model-cache\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.558613 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.558554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-tmp-dir\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.558613 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.558597    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-home\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.560682 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.560663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-dshm\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.560773 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.560742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff91499-f7bc-439f-954b-e38d933f816a-tls-certs\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.569430 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.569407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvxz\" (UniqueName: \"kubernetes.io/projected/9ff91499-f7bc-439f-954b-e38d933f816a-kube-api-access-tdvxz\") pod \"sub-test-model-85f004-kserve-7f8947b74f-mbz5x\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") " pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.638939 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.638853    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:18.777890 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.777864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"]
Jul 27 21:36:18.780036 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:36:18.780008    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ff91499_f7bc_439f_954b_e38d933f816a.slice/crio-b35445267e22a195cea49edc5e60a7f3b2e10e5c8ae2d2126905ac37f336d1aa WatchSource:0}: Error finding container b35445267e22a195cea49edc5e60a7f3b2e10e5c8ae2d2126905ac37f336d1aa: Status 404 returned error can't find the container with id b35445267e22a195cea49edc5e60a7f3b2e10e5c8ae2d2126905ac37f336d1aa
Jul 27 21:36:18.822367 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:18.822336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x" event={"ID":"9ff91499-f7bc-439f-954b-e38d933f816a","Type":"ContainerStarted","Data":"b35445267e22a195cea49edc5e60a7f3b2e10e5c8ae2d2126905ac37f336d1aa"}
Jul 27 21:36:19.829854 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:19.829821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x" event={"ID":"9ff91499-f7bc-439f-954b-e38d933f816a","Type":"ContainerStarted","Data":"ade48ac0031605e279b9b6a6c55af7ac1b8d3604659d099b2795d7ac4de018c9"}
Jul 27 21:36:19.848690 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:19.848643    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x" podStartSLOduration=1.848627 podStartE2EDuration="1.848627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:19.846347317 +0000 UTC m=+2697.211913185" watchObservedRunningTime="2026-07-27 21:36:19.848627 +0000 UTC m=+2697.214192892"
Jul 27 21:36:22.001998 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:22.001961    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:36:23.499965 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:23.499933    2577 scope.go:117] "RemoveContainer" containerID="73abcbae254137956b6a48ee965d7962856e12d7bb3671f78f1c362ee97c9361"
Jul 27 21:36:28.639454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:28.639416    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:28.639454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:28.639464    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:28.652160 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:28.652130    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:28.883451 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:28.883422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:36:54.063046 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:36:54.063005    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:37:12.894575 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:12.894542    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:15.154694 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:15.154658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:37:20.019930 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:20.019897    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"]
Jul 27 21:37:20.054520 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:20.054484    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:20.963178 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:20.963136    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"]
Jul 27 21:37:20.985133 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:20.985095    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:26.463317 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:26.463282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:37:27.658398 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:27.658361    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:37:35.039154 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.039102    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7" podUID="626fc1d4-85c1-4005-a921-73ab4f1248ee" containerName="main" containerID="cri-o://8528601d552e5238e243185e2c5876b68a7673b20f86fc16ec844d7db8bb41ea" gracePeriod=45
Jul 27 21:37:35.199587 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.199550    2577 generic.go:356] "Generic (PLEG): container finished" podID="626fc1d4-85c1-4005-a921-73ab4f1248ee" containerID="8528601d552e5238e243185e2c5876b68a7673b20f86fc16ec844d7db8bb41ea" exitCode=0
Jul 27 21:37:35.199750 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.199610    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7" event={"ID":"626fc1d4-85c1-4005-a921-73ab4f1248ee","Type":"ContainerDied","Data":"8528601d552e5238e243185e2c5876b68a7673b20f86fc16ec844d7db8bb41ea"}
Jul 27 21:37:35.302394 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.302329    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:37:35.367856 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.367821    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626fc1d4-85c1-4005-a921-73ab4f1248ee-tls-certs\") pod \"626fc1d4-85c1-4005-a921-73ab4f1248ee\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") "
Jul 27 21:37:35.367856 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.367856    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-home\") pod \"626fc1d4-85c1-4005-a921-73ab4f1248ee\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") "
Jul 27 21:37:35.368096 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.367904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-model-cache\") pod \"626fc1d4-85c1-4005-a921-73ab4f1248ee\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") "
Jul 27 21:37:35.368096 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.367959    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-dshm\") pod \"626fc1d4-85c1-4005-a921-73ab4f1248ee\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") "
Jul 27 21:37:35.368096 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.367988    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phwsv\" (UniqueName: \"kubernetes.io/projected/626fc1d4-85c1-4005-a921-73ab4f1248ee-kube-api-access-phwsv\") pod \"626fc1d4-85c1-4005-a921-73ab4f1248ee\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") "
Jul 27 21:37:35.368096 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.368024    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-tmp-dir\") pod \"626fc1d4-85c1-4005-a921-73ab4f1248ee\" (UID: \"626fc1d4-85c1-4005-a921-73ab4f1248ee\") "
Jul 27 21:37:35.368347 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.368164    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-home" (OuterVolumeSpecName: "home") pod "626fc1d4-85c1-4005-a921-73ab4f1248ee" (UID: "626fc1d4-85c1-4005-a921-73ab4f1248ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:35.368347 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.368211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "626fc1d4-85c1-4005-a921-73ab4f1248ee" (UID: "626fc1d4-85c1-4005-a921-73ab4f1248ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:35.368450 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.368357    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "626fc1d4-85c1-4005-a921-73ab4f1248ee" (UID: "626fc1d4-85c1-4005-a921-73ab4f1248ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:35.368450 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.368375    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-model-cache\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:35.368450 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.368394    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-home\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:35.370299 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.370277    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/626fc1d4-85c1-4005-a921-73ab4f1248ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "626fc1d4-85c1-4005-a921-73ab4f1248ee" (UID: "626fc1d4-85c1-4005-a921-73ab4f1248ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:37:35.370299 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.370288    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-dshm" (OuterVolumeSpecName: "dshm") pod "626fc1d4-85c1-4005-a921-73ab4f1248ee" (UID: "626fc1d4-85c1-4005-a921-73ab4f1248ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:35.370427 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.370390    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626fc1d4-85c1-4005-a921-73ab4f1248ee-kube-api-access-phwsv" (OuterVolumeSpecName: "kube-api-access-phwsv") pod "626fc1d4-85c1-4005-a921-73ab4f1248ee" (UID: "626fc1d4-85c1-4005-a921-73ab4f1248ee"). InnerVolumeSpecName "kube-api-access-phwsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:37:35.469710 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.469670    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-dshm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:35.469710 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.469703    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phwsv\" (UniqueName: \"kubernetes.io/projected/626fc1d4-85c1-4005-a921-73ab4f1248ee-kube-api-access-phwsv\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:35.469915 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.469735    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/626fc1d4-85c1-4005-a921-73ab4f1248ee-tmp-dir\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:35.469915 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.469745    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/626fc1d4-85c1-4005-a921-73ab4f1248ee-tls-certs\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:35.985364 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:35.985323    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x" podUID="9ff91499-f7bc-439f-954b-e38d933f816a" containerName="main" containerID="cri-o://ade48ac0031605e279b9b6a6c55af7ac1b8d3604659d099b2795d7ac4de018c9" gracePeriod=45
Jul 27 21:37:36.206161 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.206131    2577 generic.go:356] "Generic (PLEG): container finished" podID="9ff91499-f7bc-439f-954b-e38d933f816a" containerID="ade48ac0031605e279b9b6a6c55af7ac1b8d3604659d099b2795d7ac4de018c9" exitCode=0
Jul 27 21:37:36.206612 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.206221    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x" event={"ID":"9ff91499-f7bc-439f-954b-e38d933f816a","Type":"ContainerDied","Data":"ade48ac0031605e279b9b6a6c55af7ac1b8d3604659d099b2795d7ac4de018c9"}
Jul 27 21:37:36.207636 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.207613    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"
Jul 27 21:37:36.207714 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.207653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7" event={"ID":"626fc1d4-85c1-4005-a921-73ab4f1248ee","Type":"ContainerDied","Data":"6b2af6be405726c285563c44bc17c53ac1d185a761b911855cc25d6988ef7ca2"}
Jul 27 21:37:36.207714 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.207695    2577 scope.go:117] "RemoveContainer" containerID="8528601d552e5238e243185e2c5876b68a7673b20f86fc16ec844d7db8bb41ea"
Jul 27 21:37:36.228737 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.228712    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"]
Jul 27 21:37:36.233147 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.233127    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c910ce/sub-test-model-c910ce-kserve-9b48bf8b9-k4xg7"]
Jul 27 21:37:36.241064 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.241016    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:37:36.378141 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-home\") pod \"9ff91499-f7bc-439f-954b-e38d933f816a\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") "
Jul 27 21:37:36.378385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-dshm\") pod \"9ff91499-f7bc-439f-954b-e38d933f816a\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") "
Jul 27 21:37:36.378385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378266    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff91499-f7bc-439f-954b-e38d933f816a-tls-certs\") pod \"9ff91499-f7bc-439f-954b-e38d933f816a\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") "
Jul 27 21:37:36.378385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378297    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdvxz\" (UniqueName: \"kubernetes.io/projected/9ff91499-f7bc-439f-954b-e38d933f816a-kube-api-access-tdvxz\") pod \"9ff91499-f7bc-439f-954b-e38d933f816a\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") "
Jul 27 21:37:36.378385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378328    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-model-cache\") pod \"9ff91499-f7bc-439f-954b-e38d933f816a\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") "
Jul 27 21:37:36.378600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378393    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-tmp-dir\") pod \"9ff91499-f7bc-439f-954b-e38d933f816a\" (UID: \"9ff91499-f7bc-439f-954b-e38d933f816a\") "
Jul 27 21:37:36.378600 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378416    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-home" (OuterVolumeSpecName: "home") pod "9ff91499-f7bc-439f-954b-e38d933f816a" (UID: "9ff91499-f7bc-439f-954b-e38d933f816a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:36.378694 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ff91499-f7bc-439f-954b-e38d933f816a" (UID: "9ff91499-f7bc-439f-954b-e38d933f816a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:36.378694 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378640    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-home\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:36.378762 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.378698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ff91499-f7bc-439f-954b-e38d933f816a" (UID: "9ff91499-f7bc-439f-954b-e38d933f816a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:36.380674 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.380649    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-dshm" (OuterVolumeSpecName: "dshm") pod "9ff91499-f7bc-439f-954b-e38d933f816a" (UID: "9ff91499-f7bc-439f-954b-e38d933f816a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:37:36.380782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.380739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff91499-f7bc-439f-954b-e38d933f816a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ff91499-f7bc-439f-954b-e38d933f816a" (UID: "9ff91499-f7bc-439f-954b-e38d933f816a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:37:36.380782 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.380745    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff91499-f7bc-439f-954b-e38d933f816a-kube-api-access-tdvxz" (OuterVolumeSpecName: "kube-api-access-tdvxz") pod "9ff91499-f7bc-439f-954b-e38d933f816a" (UID: "9ff91499-f7bc-439f-954b-e38d933f816a"). InnerVolumeSpecName "kube-api-access-tdvxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:37:36.479296 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.479243    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-tmp-dir\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:36.479296 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.479291    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-dshm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:36.479296 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.479300    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ff91499-f7bc-439f-954b-e38d933f816a-tls-certs\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:36.479296 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.479310    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvxz\" (UniqueName: \"kubernetes.io/projected/9ff91499-f7bc-439f-954b-e38d933f816a-kube-api-access-tdvxz\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:36.479584 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:36.479321    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ff91499-f7bc-439f-954b-e38d933f816a-model-cache\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:37:37.185384 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:37.185346    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="626fc1d4-85c1-4005-a921-73ab4f1248ee" path="/var/lib/kubelet/pods/626fc1d4-85c1-4005-a921-73ab4f1248ee/volumes"
Jul 27 21:37:37.213931 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:37.213894    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"
Jul 27 21:37:37.213931 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:37.213922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x" event={"ID":"9ff91499-f7bc-439f-954b-e38d933f816a","Type":"ContainerDied","Data":"b35445267e22a195cea49edc5e60a7f3b2e10e5c8ae2d2126905ac37f336d1aa"}
Jul 27 21:37:37.214509 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:37.213971    2577 scope.go:117] "RemoveContainer" containerID="ade48ac0031605e279b9b6a6c55af7ac1b8d3604659d099b2795d7ac4de018c9"
Jul 27 21:37:37.231450 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:37.231422    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"]
Jul 27 21:37:37.235131 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:37.235113    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/sub-test-model-85f004-kserve-7f8947b74f-mbz5x"]
Jul 27 21:37:38.555416 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:38.555379    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:37:39.184845 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:37:39.184815    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ff91499-f7bc-439f-954b-e38d933f816a" path="/var/lib/kubelet/pods/9ff91499-f7bc-439f-954b-e38d933f816a/volumes"
Jul 27 21:38:03.328966 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.328931    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"]
Jul 27 21:38:03.329497 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.329347    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ff91499-f7bc-439f-954b-e38d933f816a" containerName="main"
Jul 27 21:38:03.329497 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.329360    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ff91499-f7bc-439f-954b-e38d933f816a" containerName="main"
Jul 27 21:38:03.329497 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.329377    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626fc1d4-85c1-4005-a921-73ab4f1248ee" containerName="main"
Jul 27 21:38:03.329497 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.329383    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="626fc1d4-85c1-4005-a921-73ab4f1248ee" containerName="main"
Jul 27 21:38:03.329497 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.329439    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ff91499-f7bc-439f-954b-e38d933f816a" containerName="main"
Jul 27 21:38:03.329497 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.329449    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="626fc1d4-85c1-4005-a921-73ab4f1248ee" containerName="main"
Jul 27 21:38:03.332313 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.332298    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.336129 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.336101    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-85f004\"/\"openshift-service-ca.crt\""
Jul 27 21:38:03.336294 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.336152    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-85f004\"/\"rate-test-model-85f004-kserve-self-signed-certs\""
Jul 27 21:38:03.336294 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.336214    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-85f004\"/\"default-dockercfg-59fsx\""
Jul 27 21:38:03.336294 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.336270    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-85f004\"/\"kube-root-ca.crt\""
Jul 27 21:38:03.341723 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.341703    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"]
Jul 27 21:38:03.352439 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.352403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:38:03.509009 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.508956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tls-certs\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.509009 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.509020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-model-cache\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.509287 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.509082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-home\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.509287 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.509116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-dshm\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.509287 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.509135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq72g\" (UniqueName: \"kubernetes.io/projected/d53fb616-03bf-4511-9a1f-c33bb3e8a419-kube-api-access-bq72g\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.509287 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.509163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tmp-dir\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.609977 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.609868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-dshm\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.609977 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.609923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq72g\" (UniqueName: \"kubernetes.io/projected/d53fb616-03bf-4511-9a1f-c33bb3e8a419-kube-api-access-bq72g\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.609977 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.609966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tmp-dir\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.610392 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.610028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tls-certs\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.610392 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.610066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-model-cache\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.610392 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.610123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-home\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.610559 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.610424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tmp-dir\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.610559 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.610429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-model-cache\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.610559 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.610510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-home\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.612316 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.612296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-dshm\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.612567 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.612549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tls-certs\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.620300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.620277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq72g\" (UniqueName: \"kubernetes.io/projected/d53fb616-03bf-4511-9a1f-c33bb3e8a419-kube-api-access-bq72g\") pod \"rate-test-model-85f004-kserve-d54696959-wvfvz\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") " pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.642849 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.642825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:03.779153 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:03.779127    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"]
Jul 27 21:38:03.780848 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:38:03.780822    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd53fb616_03bf_4511_9a1f_c33bb3e8a419.slice/crio-51017f4222731218191583d891c981306317292b7d090d9e26a3258799e50100 WatchSource:0}: Error finding container 51017f4222731218191583d891c981306317292b7d090d9e26a3258799e50100: Status 404 returned error can't find the container with id 51017f4222731218191583d891c981306317292b7d090d9e26a3258799e50100
Jul 27 21:38:04.349769 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:04.349716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz" event={"ID":"d53fb616-03bf-4511-9a1f-c33bb3e8a419","Type":"ContainerStarted","Data":"7a9cdab6269f172801abf890d128b6fa4d9599cb793a30197b03ba2ffd1c31c5"}
Jul 27 21:38:04.350164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:04.349777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz" event={"ID":"d53fb616-03bf-4511-9a1f-c33bb3e8a419","Type":"ContainerStarted","Data":"51017f4222731218191583d891c981306317292b7d090d9e26a3258799e50100"}
Jul 27 21:38:04.366618 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:04.366569    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz" podStartSLOduration=1.366555406 podStartE2EDuration="1.366555406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:38:04.365153786 +0000 UTC m=+2801.730719636" watchObservedRunningTime="2026-07-27 21:38:04.366555406 +0000 UTC m=+2801.732121253"
Jul 27 21:38:07.161377 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:07.161343    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:38:13.643818 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:13.643785    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:13.644236 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:13.643834    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:13.656315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:13.656291    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:14.409239 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:14.409214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:38:45.571156 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:45.571123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:38:55.485487 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:38:55.485452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:01.133461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:01.133421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:01.234790 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:01.234757    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:04.003258 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:04.003226    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:39:24.059513 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:24.059429    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:39:30.925710 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:30.925677    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:31.622252 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:31.622210    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"]
Jul 27 21:39:31.643983 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:31.643952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:38.501930 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:38.501891    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:39:46.641371 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:46.641325    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz" podUID="d53fb616-03bf-4511-9a1f-c33bb3e8a419" containerName="main" containerID="cri-o://7a9cdab6269f172801abf890d128b6fa4d9599cb793a30197b03ba2ffd1c31c5" gracePeriod=45
Jul 27 21:39:46.855922 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:46.855894    2577 generic.go:356] "Generic (PLEG): container finished" podID="d53fb616-03bf-4511-9a1f-c33bb3e8a419" containerID="7a9cdab6269f172801abf890d128b6fa4d9599cb793a30197b03ba2ffd1c31c5" exitCode=0
Jul 27 21:39:46.856095 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:46.855980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz" event={"ID":"d53fb616-03bf-4511-9a1f-c33bb3e8a419","Type":"ContainerDied","Data":"7a9cdab6269f172801abf890d128b6fa4d9599cb793a30197b03ba2ffd1c31c5"}
Jul 27 21:39:46.896941 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:46.896877    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:39:47.045077 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045044    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq72g\" (UniqueName: \"kubernetes.io/projected/d53fb616-03bf-4511-9a1f-c33bb3e8a419-kube-api-access-bq72g\") pod \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") "
Jul 27 21:39:47.045077 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045079    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-model-cache\") pod \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") "
Jul 27 21:39:47.045363 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045100    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tmp-dir\") pod \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") "
Jul 27 21:39:47.045363 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-home\") pod \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") "
Jul 27 21:39:47.045363 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-dshm\") pod \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") "
Jul 27 21:39:47.045363 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045219    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tls-certs\") pod \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\" (UID: \"d53fb616-03bf-4511-9a1f-c33bb3e8a419\") "
Jul 27 21:39:47.045577 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045400    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-home" (OuterVolumeSpecName: "home") pod "d53fb616-03bf-4511-9a1f-c33bb3e8a419" (UID: "d53fb616-03bf-4511-9a1f-c33bb3e8a419"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:39:47.045577 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045430    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-model-cache" (OuterVolumeSpecName: "model-cache") pod "d53fb616-03bf-4511-9a1f-c33bb3e8a419" (UID: "d53fb616-03bf-4511-9a1f-c33bb3e8a419"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:39:47.045577 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045452    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d53fb616-03bf-4511-9a1f-c33bb3e8a419" (UID: "d53fb616-03bf-4511-9a1f-c33bb3e8a419"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:39:47.045729 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045574    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tmp-dir\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:39:47.045729 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045593    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-home\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:39:47.045729 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.045607    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-model-cache\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:39:47.047465 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.047446    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-dshm" (OuterVolumeSpecName: "dshm") pod "d53fb616-03bf-4511-9a1f-c33bb3e8a419" (UID: "d53fb616-03bf-4511-9a1f-c33bb3e8a419"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:39:47.047527 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.047512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53fb616-03bf-4511-9a1f-c33bb3e8a419-kube-api-access-bq72g" (OuterVolumeSpecName: "kube-api-access-bq72g") pod "d53fb616-03bf-4511-9a1f-c33bb3e8a419" (UID: "d53fb616-03bf-4511-9a1f-c33bb3e8a419"). InnerVolumeSpecName "kube-api-access-bq72g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:39:47.047565 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.047513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d53fb616-03bf-4511-9a1f-c33bb3e8a419" (UID: "d53fb616-03bf-4511-9a1f-c33bb3e8a419"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:39:47.146536 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.146498    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53fb616-03bf-4511-9a1f-c33bb3e8a419-dshm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:39:47.146536 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.146524    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53fb616-03bf-4511-9a1f-c33bb3e8a419-tls-certs\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:39:47.146536 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.146536    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq72g\" (UniqueName: \"kubernetes.io/projected/d53fb616-03bf-4511-9a1f-c33bb3e8a419-kube-api-access-bq72g\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:39:47.390543 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.390509    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7115-140245860547584"]
Jul 27 21:39:47.390960 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.390941    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53fb616-03bf-4511-9a1f-c33bb3e8a419" containerName="main"
Jul 27 21:39:47.391041 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.390962    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53fb616-03bf-4511-9a1f-c33bb3e8a419" containerName="main"
Jul 27 21:39:47.391100 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.391088    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53fb616-03bf-4511-9a1f-c33bb3e8a419" containerName="main"
Jul 27 21:39:47.394137 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.394119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7115-140245860547584"
Jul 27 21:39:47.396672 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.396648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jq5dl\""
Jul 27 21:39:47.398435 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.398410    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7115-140245860547584"]
Jul 27 21:39:47.551333 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.551304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctgm\" (UniqueName: \"kubernetes.io/projected/1d18832c-7931-46a2-a962-e1b167c3a68a-kube-api-access-qctgm\") pod \"test-curl-7115-140245860547584\" (UID: \"1d18832c-7931-46a2-a962-e1b167c3a68a\") " pod="opendatahub/test-curl-7115-140245860547584"
Jul 27 21:39:47.652724 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.652692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctgm\" (UniqueName: \"kubernetes.io/projected/1d18832c-7931-46a2-a962-e1b167c3a68a-kube-api-access-qctgm\") pod \"test-curl-7115-140245860547584\" (UID: \"1d18832c-7931-46a2-a962-e1b167c3a68a\") " pod="opendatahub/test-curl-7115-140245860547584"
Jul 27 21:39:47.661763 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.661739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctgm\" (UniqueName: \"kubernetes.io/projected/1d18832c-7931-46a2-a962-e1b167c3a68a-kube-api-access-qctgm\") pod \"test-curl-7115-140245860547584\" (UID: \"1d18832c-7931-46a2-a962-e1b167c3a68a\") " pod="opendatahub/test-curl-7115-140245860547584"
Jul 27 21:39:47.705634 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.705604    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7115-140245860547584"
Jul 27 21:39:47.833777 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.833726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7115-140245860547584"]
Jul 27 21:39:47.836085 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:39:47.836057    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d18832c_7931_46a2_a962_e1b167c3a68a.slice/crio-6ab62381aba59e03b092eb8e7f4f686538a88abc2bd859051e5d6362a47dee7b WatchSource:0}: Error finding container 6ab62381aba59e03b092eb8e7f4f686538a88abc2bd859051e5d6362a47dee7b: Status 404 returned error can't find the container with id 6ab62381aba59e03b092eb8e7f4f686538a88abc2bd859051e5d6362a47dee7b
Jul 27 21:39:47.837536 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.837518    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:39:47.862683 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.862658    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"
Jul 27 21:39:47.862802 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.862692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz" event={"ID":"d53fb616-03bf-4511-9a1f-c33bb3e8a419","Type":"ContainerDied","Data":"51017f4222731218191583d891c981306317292b7d090d9e26a3258799e50100"}
Jul 27 21:39:47.862802 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.862739    2577 scope.go:117] "RemoveContainer" containerID="7a9cdab6269f172801abf890d128b6fa4d9599cb793a30197b03ba2ffd1c31c5"
Jul 27 21:39:47.863967 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.863937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7115-140245860547584" event={"ID":"1d18832c-7931-46a2-a962-e1b167c3a68a","Type":"ContainerStarted","Data":"6ab62381aba59e03b092eb8e7f4f686538a88abc2bd859051e5d6362a47dee7b"}
Jul 27 21:39:47.881562 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.881541    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"]
Jul 27 21:39:47.886592 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:47.886570    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-85f004/rate-test-model-85f004-kserve-d54696959-wvfvz"]
Jul 27 21:39:48.871162 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:48.871070    2577 generic.go:356] "Generic (PLEG): container finished" podID="1d18832c-7931-46a2-a962-e1b167c3a68a" containerID="f9358a73335b8a5646c3731c92e5eaa720085ace79b6a87c02c4a50239b95845" exitCode=0
Jul 27 21:39:48.871162 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:48.871117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7115-140245860547584" event={"ID":"1d18832c-7931-46a2-a962-e1b167c3a68a","Type":"ContainerDied","Data":"f9358a73335b8a5646c3731c92e5eaa720085ace79b6a87c02c4a50239b95845"}
Jul 27 21:39:49.185110 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:49.185030    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d53fb616-03bf-4511-9a1f-c33bb3e8a419" path="/var/lib/kubelet/pods/d53fb616-03bf-4511-9a1f-c33bb3e8a419/volumes"
Jul 27 21:39:50.008992 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.008966    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7115-140245860547584"
Jul 27 21:39:50.028945 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.028924    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7115-140245860547584_1d18832c-7931-46a2-a962-e1b167c3a68a/test-curl-7115-140245860547584/0.log"
Jul 27 21:39:50.050850 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.050820    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7115-140245860547584"]
Jul 27 21:39:50.053645 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.053622    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7115-140245860547584"]
Jul 27 21:39:50.071100 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.071078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qctgm\" (UniqueName: \"kubernetes.io/projected/1d18832c-7931-46a2-a962-e1b167c3a68a-kube-api-access-qctgm\") pod \"1d18832c-7931-46a2-a962-e1b167c3a68a\" (UID: \"1d18832c-7931-46a2-a962-e1b167c3a68a\") "
Jul 27 21:39:50.073326 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.073307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d18832c-7931-46a2-a962-e1b167c3a68a-kube-api-access-qctgm" (OuterVolumeSpecName: "kube-api-access-qctgm") pod "1d18832c-7931-46a2-a962-e1b167c3a68a" (UID: "1d18832c-7931-46a2-a962-e1b167c3a68a"). InnerVolumeSpecName "kube-api-access-qctgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:39:50.178143 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.172556    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qctgm\" (UniqueName: \"kubernetes.io/projected/1d18832c-7931-46a2-a962-e1b167c3a68a-kube-api-access-qctgm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:39:50.545150 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.545121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:50.862660 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.862583    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:50.883229 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.883201    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7115-140245860547584"
Jul 27 21:39:50.883412 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:50.883205    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ab62381aba59e03b092eb8e7f4f686538a88abc2bd859051e5d6362a47dee7b"
Jul 27 21:39:50.895994 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:39:50.895966    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7115-140245860547584\" is forbidden: User \"system:node:ip-10-0-131-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-91.ec2.internal' and this object" podUID="1d18832c-7931-46a2-a962-e1b167c3a68a" pod="opendatahub/test-curl-7115-140245860547584"
Jul 27 21:39:51.182826 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:51.182737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:51.185574 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:51.185550    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d18832c-7931-46a2-a962-e1b167c3a68a" path="/var/lib/kubelet/pods/1d18832c-7931-46a2-a962-e1b167c3a68a/volumes"
Jul 27 21:39:54.332395 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.332359    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:54.431899 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.431870    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:54.433433 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.433408    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"]
Jul 27 21:39:54.433942 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.433917    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d18832c-7931-46a2-a962-e1b167c3a68a" containerName="test-curl-7115-140245860547584"
Jul 27 21:39:54.433942 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.433941    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d18832c-7931-46a2-a962-e1b167c3a68a" containerName="test-curl-7115-140245860547584"
Jul 27 21:39:54.434086 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.434044    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d18832c-7931-46a2-a962-e1b167c3a68a" containerName="test-curl-7115-140245860547584"
Jul 27 21:39:54.437381 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.437365    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.439903 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.439884    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5d8041-serving-cert\""
Jul 27 21:39:54.447573 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.447542    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"]
Jul 27 21:39:54.507090 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.507054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66cf0f51-5be7-4818-acbc-4877db8e9604-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.507300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.507116    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ng4\" (UniqueName: \"kubernetes.io/projected/66cf0f51-5be7-4818-acbc-4877db8e9604-kube-api-access-g5ng4\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.507300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.507192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66cf0f51-5be7-4818-acbc-4877db8e9604-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.569060 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.569030    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:54.608720 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.608644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66cf0f51-5be7-4818-acbc-4877db8e9604-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.608720 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.608704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ng4\" (UniqueName: \"kubernetes.io/projected/66cf0f51-5be7-4818-acbc-4877db8e9604-kube-api-access-g5ng4\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.608899 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.608744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66cf0f51-5be7-4818-acbc-4877db8e9604-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.609430 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.609412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66cf0f51-5be7-4818-acbc-4877db8e9604-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.611409 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.611388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66cf0f51-5be7-4818-acbc-4877db8e9604-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.621070 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.621045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ng4\" (UniqueName: \"kubernetes.io/projected/66cf0f51-5be7-4818-acbc-4877db8e9604-kube-api-access-g5ng4\") pod \"maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.668967 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.668924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:54.749446 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.749415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:54.882730 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.882706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"]
Jul 27 21:39:54.885442 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:39:54.885403    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cf0f51_5be7_4818_acbc_4877db8e9604.slice/crio-d7ae2ac4fa1f8f2b714fd6261fa28be79de431bb4c15d31f112ad3fc31b64013 WatchSource:0}: Error finding container d7ae2ac4fa1f8f2b714fd6261fa28be79de431bb4c15d31f112ad3fc31b64013: Status 404 returned error can't find the container with id d7ae2ac4fa1f8f2b714fd6261fa28be79de431bb4c15d31f112ad3fc31b64013
Jul 27 21:39:54.904666 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:54.904639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx" event={"ID":"66cf0f51-5be7-4818-acbc-4877db8e9604","Type":"ContainerStarted","Data":"d7ae2ac4fa1f8f2b714fd6261fa28be79de431bb4c15d31f112ad3fc31b64013"}
Jul 27 21:39:55.041373 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.041335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:55.137883 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.137802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:39:55.225346 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.225313    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"]
Jul 27 21:39:55.230216 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.230176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.232871 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.232848    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5d8041-dockercfg-nzmfh\""
Jul 27 21:39:55.233253 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.232850    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5d8041\""
Jul 27 21:39:55.233777 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.233752    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"]
Jul 27 21:39:55.314970 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.314932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shf29\" (UniqueName: \"kubernetes.io/projected/50824431-23ee-4d62-876f-049811b78db9-kube-api-access-shf29\") pod \"payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5\" (UID: \"50824431-23ee-4d62-876f-049811b78db9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.315155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.314998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50824431-23ee-4d62-876f-049811b78db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5\" (UID: \"50824431-23ee-4d62-876f-049811b78db9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.416494 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.416391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50824431-23ee-4d62-876f-049811b78db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5\" (UID: \"50824431-23ee-4d62-876f-049811b78db9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.416954 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.416525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shf29\" (UniqueName: \"kubernetes.io/projected/50824431-23ee-4d62-876f-049811b78db9-kube-api-access-shf29\") pod \"payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5\" (UID: \"50824431-23ee-4d62-876f-049811b78db9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.417064 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.417043    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50824431-23ee-4d62-876f-049811b78db9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5\" (UID: \"50824431-23ee-4d62-876f-049811b78db9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.429110 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.429079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shf29\" (UniqueName: \"kubernetes.io/projected/50824431-23ee-4d62-876f-049811b78db9-kube-api-access-shf29\") pod \"payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5\" (UID: \"50824431-23ee-4d62-876f-049811b78db9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.541936 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.541896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.685006 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.684977    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"]
Jul 27 21:39:55.700421 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:39:55.700394    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50824431_23ee_4d62_876f_049811b78db9.slice/crio-ef50370866ad14d7cbd79ff1b7b324fc895ae1e3ba73935e5b3076a52aec8c2f WatchSource:0}: Error finding container ef50370866ad14d7cbd79ff1b7b324fc895ae1e3ba73935e5b3076a52aec8c2f: Status 404 returned error can't find the container with id ef50370866ad14d7cbd79ff1b7b324fc895ae1e3ba73935e5b3076a52aec8c2f
Jul 27 21:39:55.912914 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.912877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx" event={"ID":"66cf0f51-5be7-4818-acbc-4877db8e9604","Type":"ContainerStarted","Data":"141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937"}
Jul 27 21:39:55.913125 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.912999    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:39:55.914655 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.914628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5" event={"ID":"50824431-23ee-4d62-876f-049811b78db9","Type":"ContainerStarted","Data":"6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3"}
Jul 27 21:39:55.914770 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.914672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5" event={"ID":"50824431-23ee-4d62-876f-049811b78db9","Type":"ContainerStarted","Data":"ef50370866ad14d7cbd79ff1b7b324fc895ae1e3ba73935e5b3076a52aec8c2f"}
Jul 27 21:39:55.914770 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.914698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:39:55.930177 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.930084    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx" podStartSLOduration=1.155621276 podStartE2EDuration="1.930069787s" totalImagesPullingTime="774.448511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:55.928356036 +0000 UTC m=+2913.293921884" watchObservedRunningTime="2026-07-27 21:39:55.930069787 +0000 UTC m=+2913.295635638"
Jul 27 21:39:55.944972 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:55.944920    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5" podStartSLOduration=0.944903908 podStartE2EDuration="944.903908ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:39:55.943801607 +0000 UTC m=+2913.309367458" watchObservedRunningTime="2026-07-27 21:39:55.944903908 +0000 UTC m=+2913.310469802"
Jul 27 21:39:58.798032 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:39:58.798001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:01.926520 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:01.926490    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:40:03.154159 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:03.154120    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:40:06.922706 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:06.922675    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:40:09.676093 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:09.676054    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:09.973522 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:09.973442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:10.331516 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:10.331481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:16.135975 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.135939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:16.237656 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.237610    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:16.464415 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.464336    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"]
Jul 27 21:40:16.468786 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.468764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.471486 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.471462    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0e6399-serving-cert\""
Jul 27 21:40:16.480748 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.480715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"]
Jul 27 21:40:16.500721 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.500678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:16.608930 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.608901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a1057c-36be-4dc7-8411-dc75dd416e3f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.609101 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.608959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a1057c-36be-4dc7-8411-dc75dd416e3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.609101 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.609045    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzr2\" (UniqueName: \"kubernetes.io/projected/58a1057c-36be-4dc7-8411-dc75dd416e3f-kube-api-access-nvzr2\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.620765 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.620733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:16.709813 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.709773    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a1057c-36be-4dc7-8411-dc75dd416e3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.709984 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.709821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzr2\" (UniqueName: \"kubernetes.io/projected/58a1057c-36be-4dc7-8411-dc75dd416e3f-kube-api-access-nvzr2\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.709984 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.709889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a1057c-36be-4dc7-8411-dc75dd416e3f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.710530 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.710510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a1057c-36be-4dc7-8411-dc75dd416e3f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.712545 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.712521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a1057c-36be-4dc7-8411-dc75dd416e3f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.719418 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.719362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzr2\" (UniqueName: \"kubernetes.io/projected/58a1057c-36be-4dc7-8411-dc75dd416e3f-kube-api-access-nvzr2\") pod \"maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.782250 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.782222    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:16.916374 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:16.916345    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"]
Jul 27 21:40:16.919002 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:40:16.918975    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a1057c_36be_4dc7_8411_dc75dd416e3f.slice/crio-a08d021344c45f6b9825cc2b28b4078a88c85b8565c4defa8ee7ed5fc9ace83f WatchSource:0}: Error finding container a08d021344c45f6b9825cc2b28b4078a88c85b8565c4defa8ee7ed5fc9ace83f: Status 404 returned error can't find the container with id a08d021344c45f6b9825cc2b28b4078a88c85b8565c4defa8ee7ed5fc9ace83f
Jul 27 21:40:17.009926 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.009890    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:17.028487 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.028446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn" event={"ID":"58a1057c-36be-4dc7-8411-dc75dd416e3f","Type":"ContainerStarted","Data":"a08d021344c45f6b9825cc2b28b4078a88c85b8565c4defa8ee7ed5fc9ace83f"}
Jul 27 21:40:17.107853 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.107819    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:17.217144 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.217113    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"]
Jul 27 21:40:17.221770 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.221751    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:17.225011 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.224982    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0e6399-dockercfg-kwdxt\""
Jul 27 21:40:17.225011 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.224999    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0e6399\""
Jul 27 21:40:17.234494 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.234475    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"]
Jul 27 21:40:17.316269 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.316168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15abb7d6-a85e-4017-9de4-54ca20fdfe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv\" (UID: \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:17.316444 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.316288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72lfj\" (UniqueName: \"kubernetes.io/projected/15abb7d6-a85e-4017-9de4-54ca20fdfe64-kube-api-access-72lfj\") pod \"payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv\" (UID: \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:17.417112 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.417074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72lfj\" (UniqueName: \"kubernetes.io/projected/15abb7d6-a85e-4017-9de4-54ca20fdfe64-kube-api-access-72lfj\") pod \"payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv\" (UID: \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:17.417291 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.417228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15abb7d6-a85e-4017-9de4-54ca20fdfe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv\" (UID: \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:17.418004 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.417986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15abb7d6-a85e-4017-9de4-54ca20fdfe64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv\" (UID: \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:17.426875 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.426848    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72lfj\" (UniqueName: \"kubernetes.io/projected/15abb7d6-a85e-4017-9de4-54ca20fdfe64-kube-api-access-72lfj\") pod \"payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv\" (UID: \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:17.531906 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.531883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:17.670339 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:17.670308    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"]
Jul 27 21:40:17.673694 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:40:17.673659    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15abb7d6_a85e_4017_9de4_54ca20fdfe64.slice/crio-1611a0d66fd614b96d06752703431455922658ef993b0ec8d1d92b5170302bf8 WatchSource:0}: Error finding container 1611a0d66fd614b96d06752703431455922658ef993b0ec8d1d92b5170302bf8: Status 404 returned error can't find the container with id 1611a0d66fd614b96d06752703431455922658ef993b0ec8d1d92b5170302bf8
Jul 27 21:40:18.035561 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:18.035522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv" event={"ID":"15abb7d6-a85e-4017-9de4-54ca20fdfe64","Type":"ContainerStarted","Data":"79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc"}
Jul 27 21:40:18.035772 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:18.035587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv" event={"ID":"15abb7d6-a85e-4017-9de4-54ca20fdfe64","Type":"ContainerStarted","Data":"1611a0d66fd614b96d06752703431455922658ef993b0ec8d1d92b5170302bf8"}
Jul 27 21:40:18.035772 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:18.035717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:18.037031 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:18.037006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn" event={"ID":"58a1057c-36be-4dc7-8411-dc75dd416e3f","Type":"ContainerStarted","Data":"90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116"}
Jul 27 21:40:18.037174 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:18.037056    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:18.054831 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:18.054790    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv" podStartSLOduration=1.054776382 podStartE2EDuration="1.054776382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:18.052766198 +0000 UTC m=+2935.418332046" watchObservedRunningTime="2026-07-27 21:40:18.054776382 +0000 UTC m=+2935.420342229"
Jul 27 21:40:18.076916 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:18.076868    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn" podStartSLOduration=1.533853586 podStartE2EDuration="2.076854942s" totalImagesPullingTime="543.001356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:18.07472124 +0000 UTC m=+2935.440287086" watchObservedRunningTime="2026-07-27 21:40:18.076854942 +0000 UTC m=+2935.442420828"
Jul 27 21:40:20.616859 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:20.616818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:23.571904 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:23.571874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:40:24.047506 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:24.047477    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:40:29.044566 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:29.044538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:40:44.729908 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.729831    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"]
Jul 27 21:40:44.733323 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.733306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.736549 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.736528    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5d8041\"/\"openshift-service-ca.crt\""
Jul 27 21:40:44.737605 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.737586    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-5d8041\"/\"kube-root-ca.crt\""
Jul 27 21:40:44.737700 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.737644    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5d8041\"/\"ipp-route-5d8041-kserve-self-signed-certs\""
Jul 27 21:40:44.737700 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.737649    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-5d8041\"/\"default-dockercfg-ppcfb\""
Jul 27 21:40:44.745668 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.745645    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"]
Jul 27 21:40:44.753784 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.753761    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-model-cache\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.753913 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.753791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-home\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.753913 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.753861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-dshm\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.753913 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.753884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4297592-64e6-4022-867f-d46cb87764a7-tls-certs\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.754077 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.753923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwzm\" (UniqueName: \"kubernetes.io/projected/d4297592-64e6-4022-867f-d46cb87764a7-kube-api-access-zjwzm\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.754077 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.753956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-tmp-dir\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.796401 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.796355    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:40:44.854386 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854351    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-model-cache\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.854386 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-home\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.854654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-dshm\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.854654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4297592-64e6-4022-867f-d46cb87764a7-tls-certs\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.854654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwzm\" (UniqueName: \"kubernetes.io/projected/d4297592-64e6-4022-867f-d46cb87764a7-kube-api-access-zjwzm\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.854654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-tmp-dir\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.854861 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-model-cache\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.854861 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-home\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.854975 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.854887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-tmp-dir\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.856933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.856912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-dshm\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.857101 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.857082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4297592-64e6-4022-867f-d46cb87764a7-tls-certs\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:44.864813 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:44.864789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwzm\" (UniqueName: \"kubernetes.io/projected/d4297592-64e6-4022-867f-d46cb87764a7-kube-api-access-zjwzm\") pod \"ipp-route-5d8041-kserve-77bd598d67-sjsxl\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") " pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:45.042936 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:45.042846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:45.183667 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:40:45.183634    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4297592_64e6_4022_867f_d46cb87764a7.slice/crio-da1beb5a3574cb9a8da8bb0e7d45dbac506904808a0361777e8bae8f1ee29639 WatchSource:0}: Error finding container da1beb5a3574cb9a8da8bb0e7d45dbac506904808a0361777e8bae8f1ee29639: Status 404 returned error can't find the container with id da1beb5a3574cb9a8da8bb0e7d45dbac506904808a0361777e8bae8f1ee29639
Jul 27 21:40:45.185328 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:45.185307    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"]
Jul 27 21:40:46.181578 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:46.181544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl" event={"ID":"d4297592-64e6-4022-867f-d46cb87764a7","Type":"ContainerStarted","Data":"9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692"}
Jul 27 21:40:46.181961 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:46.181586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl" event={"ID":"d4297592-64e6-4022-867f-d46cb87764a7","Type":"ContainerStarted","Data":"da1beb5a3574cb9a8da8bb0e7d45dbac506904808a0361777e8bae8f1ee29639"}
Jul 27 21:40:48.652496 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:48.652432    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl" podStartSLOduration=4.652415178 podStartE2EDuration="4.652415178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:46.220373747 +0000 UTC m=+2963.585939594" watchObservedRunningTime="2026-07-27 21:40:48.652415178 +0000 UTC m=+2966.017981024"
Jul 27 21:40:48.654058 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:48.654027    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:40:55.043664 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:55.043626    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:55.044225 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:55.043678    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:55.056747 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:55.056719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:40:55.234910 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:40:55.234883    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:41:23.580139 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:23.580099    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:41:36.498895 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:36.498858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:36.860129 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:36.860049    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:37.076102 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:37.076057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:39.976847 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:39.976811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:40.161557 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:40.161525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:41.718387 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.718354    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"]
Jul 27 21:41:41.722121 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.722095    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:41.726434 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.726402    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ef0509-serving-cert\""
Jul 27 21:41:41.735081 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.735054    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"]
Jul 27 21:41:41.773487 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.773446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:41.776579 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.776558    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8tlp\" (UniqueName: \"kubernetes.io/projected/3ebc46c9-bd63-4099-9047-2230f599af74-kube-api-access-d8tlp\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:41.776665 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.776595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc46c9-bd63-4099-9047-2230f599af74-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:41.776665 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.776657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:41.877477 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.877440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:41.877686 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.877523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8tlp\" (UniqueName: \"kubernetes.io/projected/3ebc46c9-bd63-4099-9047-2230f599af74-kube-api-access-d8tlp\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:41.877686 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.877551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc46c9-bd63-4099-9047-2230f599af74-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:41.877686 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:41:41.877594    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-serving-cert: secret "maas-api-e2e-ipp-cleanup-ef0509-serving-cert" not found
Jul 27 21:41:41.877686 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:41:41.877665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls podName:3ebc46c9-bd63-4099-9047-2230f599af74 nodeName:}" failed. No retries permitted until 2026-07-27 21:41:42.377644031 +0000 UTC m=+3019.743209859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq" (UID: "3ebc46c9-bd63-4099-9047-2230f599af74") : secret "maas-api-e2e-ipp-cleanup-ef0509-serving-cert" not found
Jul 27 21:41:41.878160 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.878142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc46c9-bd63-4099-9047-2230f599af74-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:41.894765 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:41.894734    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8tlp\" (UniqueName: \"kubernetes.io/projected/3ebc46c9-bd63-4099-9047-2230f599af74-kube-api-access-d8tlp\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:42.021497 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:42.021462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:42.382419 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:42.382334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:42.385042 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:42.385012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:42.392496 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:42.392462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:42.492026 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:42.491977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:42.634139 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:42.634051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:42.780305 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:42.780279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"]
Jul 27 21:41:42.781998 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:41:42.781968    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebc46c9_bd63_4099_9047_2230f599af74.slice/crio-e320498ebf1e967089a6c09628757310e5a852f4f2dabb15dd89b40acbaceed5 WatchSource:0}: Error finding container e320498ebf1e967089a6c09628757310e5a852f4f2dabb15dd89b40acbaceed5: Status 404 returned error can't find the container with id e320498ebf1e967089a6c09628757310e5a852f4f2dabb15dd89b40acbaceed5
Jul 27 21:41:43.471731 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:43.471625    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq" event={"ID":"3ebc46c9-bd63-4099-9047-2230f599af74","Type":"ContainerStarted","Data":"fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332"}
Jul 27 21:41:43.471731 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:43.471686    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq" event={"ID":"3ebc46c9-bd63-4099-9047-2230f599af74","Type":"ContainerStarted","Data":"e320498ebf1e967089a6c09628757310e5a852f4f2dabb15dd89b40acbaceed5"}
Jul 27 21:41:43.471731 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:43.471726    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:43.491091 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:43.491042    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq" podStartSLOduration=2.104560365 podStartE2EDuration="2.491025782s" totalImagesPullingTime="386.465417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:41:43.490646029 +0000 UTC m=+3020.856211879" watchObservedRunningTime="2026-07-27 21:41:43.491025782 +0000 UTC m=+3020.856591656"
Jul 27 21:41:44.340287 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:44.340250    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:49.482155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:49.482125    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:41:50.102451 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:50.102414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:50.209863 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:50.209831    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:41:51.155542 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:51.155493    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:41:58.636463 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:41:58.636424    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:13.216024 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.215816    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"]
Jul 27 21:42:13.216580 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.216072    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq" podUID="3ebc46c9-bd63-4099-9047-2230f599af74" containerName="maas-api" containerID="cri-o://fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332" gracePeriod=30
Jul 27 21:42:13.243005 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.242969    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:13.322202 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.322123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:13.386328 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.386295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:13.468053 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.467998    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:42:13.485003 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.484973    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc46c9-bd63-4099-9047-2230f599af74-openshift-service-ca\") pod \"3ebc46c9-bd63-4099-9047-2230f599af74\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") "
Jul 27 21:42:13.485171 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.485011    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls\") pod \"3ebc46c9-bd63-4099-9047-2230f599af74\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") "
Jul 27 21:42:13.485171 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.485097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8tlp\" (UniqueName: \"kubernetes.io/projected/3ebc46c9-bd63-4099-9047-2230f599af74-kube-api-access-d8tlp\") pod \"3ebc46c9-bd63-4099-9047-2230f599af74\" (UID: \"3ebc46c9-bd63-4099-9047-2230f599af74\") "
Jul 27 21:42:13.485716 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.485481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ebc46c9-bd63-4099-9047-2230f599af74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ebc46c9-bd63-4099-9047-2230f599af74" (UID: "3ebc46c9-bd63-4099-9047-2230f599af74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:13.487789 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.487757    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ebc46c9-bd63-4099-9047-2230f599af74" (UID: "3ebc46c9-bd63-4099-9047-2230f599af74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:42:13.488303 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.488278    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ebc46c9-bd63-4099-9047-2230f599af74-kube-api-access-d8tlp" (OuterVolumeSpecName: "kube-api-access-d8tlp") pod "3ebc46c9-bd63-4099-9047-2230f599af74" (UID: "3ebc46c9-bd63-4099-9047-2230f599af74"). InnerVolumeSpecName "kube-api-access-d8tlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:42:13.585989 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.585950    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ebc46c9-bd63-4099-9047-2230f599af74-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:42:13.585989 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.585981    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ebc46c9-bd63-4099-9047-2230f599af74-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:42:13.585989 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.585991    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8tlp\" (UniqueName: \"kubernetes.io/projected/3ebc46c9-bd63-4099-9047-2230f599af74-kube-api-access-d8tlp\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:42:13.627058 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.627015    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ebc46c9-bd63-4099-9047-2230f599af74" containerID="fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332" exitCode=0
Jul 27 21:42:13.627233 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.627214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq" event={"ID":"3ebc46c9-bd63-4099-9047-2230f599af74","Type":"ContainerDied","Data":"fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332"}
Jul 27 21:42:13.627289 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.627275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq" event={"ID":"3ebc46c9-bd63-4099-9047-2230f599af74","Type":"ContainerDied","Data":"e320498ebf1e967089a6c09628757310e5a852f4f2dabb15dd89b40acbaceed5"}
Jul 27 21:42:13.627340 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.627315    2577 scope.go:117] "RemoveContainer" containerID="fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332"
Jul 27 21:42:13.627479 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.627423    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"
Jul 27 21:42:13.638866 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.638838    2577 scope.go:117] "RemoveContainer" containerID="fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332"
Jul 27 21:42:13.639169 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:42:13.639147    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332\": container with ID starting with fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332 not found: ID does not exist" containerID="fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332"
Jul 27 21:42:13.639300 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.639176    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332"} err="failed to get container status \"fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332\": rpc error: code = NotFound desc = could not find container \"fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332\": container with ID starting with fce561be7e5fd0dff513d61d50e3d83b8831ef070bbff8ead2b956d40782d332 not found: ID does not exist"
Jul 27 21:42:13.649419 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.649395    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"]
Jul 27 21:42:13.652952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:13.652930    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ef0509-6b6bbcdbc9-gjvcq"]
Jul 27 21:42:15.185403 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:15.185370    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ebc46c9-bd63-4099-9047-2230f599af74" path="/var/lib/kubelet/pods/3ebc46c9-bd63-4099-9047-2230f599af74/volumes"
Jul 27 21:42:15.442119 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:15.442046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:15.820954 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:15.820917    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:18.660937 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:18.660906    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:42:22.001476 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:22.001440    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:22.099022 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:22.098993    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:22.257157 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:22.257076    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:22.467781 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:22.467747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:41.869475 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:41.869431    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:42.063408 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:42.063374    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:42:42.802778 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:42.802744    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"]
Jul 27 21:42:42.803015 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:42.802986    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx" podUID="66cf0f51-5be7-4818-acbc-4877db8e9604" containerName="maas-api" containerID="cri-o://141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937" gracePeriod=30
Jul 27 21:42:42.826139 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:42.826094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:42.867966 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:42.867930    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"]
Jul 27 21:42:42.868275 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:42.868226    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5" podUID="50824431-23ee-4d62-876f-049811b78db9" containerName="payload-pre-processing" containerID="cri-o://6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3" gracePeriod=30
Jul 27 21:42:42.897368 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:42.897339    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:42.998882 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:42.998836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:43.122805 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.122776    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:42:43.159409 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.159381    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:42:43.269829 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.269790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50824431-23ee-4d62-876f-049811b78db9-plugins-config-volume\") pod \"50824431-23ee-4d62-876f-049811b78db9\" (UID: \"50824431-23ee-4d62-876f-049811b78db9\") "
Jul 27 21:42:43.269829 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.269827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66cf0f51-5be7-4818-acbc-4877db8e9604-maas-api-tls\") pod \"66cf0f51-5be7-4818-acbc-4877db8e9604\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") "
Jul 27 21:42:43.270039 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.269851    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shf29\" (UniqueName: \"kubernetes.io/projected/50824431-23ee-4d62-876f-049811b78db9-kube-api-access-shf29\") pod \"50824431-23ee-4d62-876f-049811b78db9\" (UID: \"50824431-23ee-4d62-876f-049811b78db9\") "
Jul 27 21:42:43.270039 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.269943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5ng4\" (UniqueName: \"kubernetes.io/projected/66cf0f51-5be7-4818-acbc-4877db8e9604-kube-api-access-g5ng4\") pod \"66cf0f51-5be7-4818-acbc-4877db8e9604\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") "
Jul 27 21:42:43.270039 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.270029    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66cf0f51-5be7-4818-acbc-4877db8e9604-openshift-service-ca\") pod \"66cf0f51-5be7-4818-acbc-4877db8e9604\" (UID: \"66cf0f51-5be7-4818-acbc-4877db8e9604\") "
Jul 27 21:42:43.270254 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.270230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50824431-23ee-4d62-876f-049811b78db9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50824431-23ee-4d62-876f-049811b78db9" (UID: "50824431-23ee-4d62-876f-049811b78db9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:43.270452 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.270427    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66cf0f51-5be7-4818-acbc-4877db8e9604-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66cf0f51-5be7-4818-acbc-4877db8e9604" (UID: "66cf0f51-5be7-4818-acbc-4877db8e9604"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:43.270689 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.270668    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50824431-23ee-4d62-876f-049811b78db9-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:42:43.272080 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.272057    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cf0f51-5be7-4818-acbc-4877db8e9604-kube-api-access-g5ng4" (OuterVolumeSpecName: "kube-api-access-g5ng4") pod "66cf0f51-5be7-4818-acbc-4877db8e9604" (UID: "66cf0f51-5be7-4818-acbc-4877db8e9604"). InnerVolumeSpecName "kube-api-access-g5ng4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:42:43.272177 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.272066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66cf0f51-5be7-4818-acbc-4877db8e9604-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66cf0f51-5be7-4818-acbc-4877db8e9604" (UID: "66cf0f51-5be7-4818-acbc-4877db8e9604"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:42:43.272177 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.272120    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50824431-23ee-4d62-876f-049811b78db9-kube-api-access-shf29" (OuterVolumeSpecName: "kube-api-access-shf29") pod "50824431-23ee-4d62-876f-049811b78db9" (UID: "50824431-23ee-4d62-876f-049811b78db9"). InnerVolumeSpecName "kube-api-access-shf29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:42:43.371987 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.371902    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66cf0f51-5be7-4818-acbc-4877db8e9604-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:42:43.371987 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.371931    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shf29\" (UniqueName: \"kubernetes.io/projected/50824431-23ee-4d62-876f-049811b78db9-kube-api-access-shf29\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:42:43.371987 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.371943    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ng4\" (UniqueName: \"kubernetes.io/projected/66cf0f51-5be7-4818-acbc-4877db8e9604-kube-api-access-g5ng4\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:42:43.371987 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.371952    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66cf0f51-5be7-4818-acbc-4877db8e9604-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:42:43.782271 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.782228    2577 generic.go:356] "Generic (PLEG): container finished" podID="50824431-23ee-4d62-876f-049811b78db9" containerID="6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3" exitCode=0
Jul 27 21:42:43.782445 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.782277    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"
Jul 27 21:42:43.782445 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.782328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5" event={"ID":"50824431-23ee-4d62-876f-049811b78db9","Type":"ContainerDied","Data":"6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3"}
Jul 27 21:42:43.782445 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.782384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5" event={"ID":"50824431-23ee-4d62-876f-049811b78db9","Type":"ContainerDied","Data":"ef50370866ad14d7cbd79ff1b7b324fc895ae1e3ba73935e5b3076a52aec8c2f"}
Jul 27 21:42:43.782445 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.782407    2577 scope.go:117] "RemoveContainer" containerID="6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3"
Jul 27 21:42:43.783649 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.783627    2577 generic.go:356] "Generic (PLEG): container finished" podID="66cf0f51-5be7-4818-acbc-4877db8e9604" containerID="141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937" exitCode=0
Jul 27 21:42:43.783741 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.783656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx" event={"ID":"66cf0f51-5be7-4818-acbc-4877db8e9604","Type":"ContainerDied","Data":"141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937"}
Jul 27 21:42:43.783741 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.783696    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"
Jul 27 21:42:43.783741 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.783708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx" event={"ID":"66cf0f51-5be7-4818-acbc-4877db8e9604","Type":"ContainerDied","Data":"d7ae2ac4fa1f8f2b714fd6261fa28be79de431bb4c15d31f112ad3fc31b64013"}
Jul 27 21:42:43.793470 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.793446    2577 scope.go:117] "RemoveContainer" containerID="6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3"
Jul 27 21:42:43.793767 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:42:43.793747    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3\": container with ID starting with 6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3 not found: ID does not exist" containerID="6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3"
Jul 27 21:42:43.793842 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.793780    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3"} err="failed to get container status \"6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3\": rpc error: code = NotFound desc = could not find container \"6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3\": container with ID starting with 6c90b3ca52e39d0ef8e496cc1bb19214d1abfc83b447e8b8b79d9db9ede04bc3 not found: ID does not exist"
Jul 27 21:42:43.793842 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.793805    2577 scope.go:117] "RemoveContainer" containerID="141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937"
Jul 27 21:42:43.802165 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.802090    2577 scope.go:117] "RemoveContainer" containerID="141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937"
Jul 27 21:42:43.802418 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:42:43.802395    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937\": container with ID starting with 141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937 not found: ID does not exist" containerID="141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937"
Jul 27 21:42:43.802514 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.802425    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937"} err="failed to get container status \"141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937\": rpc error: code = NotFound desc = could not find container \"141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937\": container with ID starting with 141016f043906dcba6f64e7cf46b3c7e4a1753a5b22f05e3cc3c5f50b1cdd937 not found: ID does not exist"
Jul 27 21:42:43.805265 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.805244    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"]
Jul 27 21:42:43.810855 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.810838    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5d8041-5c54958876-lqkd5"]
Jul 27 21:42:43.821933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.821916    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"]
Jul 27 21:42:43.826527 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:43.826507    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d8041-f66b7b59f-789hx"]
Jul 27 21:42:45.185101 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:45.185057    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50824431-23ee-4d62-876f-049811b78db9" path="/var/lib/kubelet/pods/50824431-23ee-4d62-876f-049811b78db9/volumes"
Jul 27 21:42:45.185665 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:45.185644    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66cf0f51-5be7-4818-acbc-4877db8e9604" path="/var/lib/kubelet/pods/66cf0f51-5be7-4818-acbc-4877db8e9604/volumes"
Jul 27 21:42:45.891791 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:45.891757    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:46.261041 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:46.261006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:51.240561 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:51.240524    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"]
Jul 27 21:42:51.339097 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:51.339063    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:42:53.155124 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:53.155088    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:42:59.975009 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:42:59.974976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:06.260271 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.260229    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl" podUID="d4297592-64e6-4022-867f-d46cb87764a7" containerName="main" containerID="cri-o://9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692" gracePeriod=45
Jul 27 21:43:06.528139 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.528111    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:43:06.590402 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590372    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-home\") pod \"d4297592-64e6-4022-867f-d46cb87764a7\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") "
Jul 27 21:43:06.590564 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-dshm\") pod \"d4297592-64e6-4022-867f-d46cb87764a7\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") "
Jul 27 21:43:06.590564 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590491    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-tmp-dir\") pod \"d4297592-64e6-4022-867f-d46cb87764a7\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") "
Jul 27 21:43:06.590564 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjwzm\" (UniqueName: \"kubernetes.io/projected/d4297592-64e6-4022-867f-d46cb87764a7-kube-api-access-zjwzm\") pod \"d4297592-64e6-4022-867f-d46cb87764a7\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") "
Jul 27 21:43:06.590727 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4297592-64e6-4022-867f-d46cb87764a7-tls-certs\") pod \"d4297592-64e6-4022-867f-d46cb87764a7\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") "
Jul 27 21:43:06.590727 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590617    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-model-cache\") pod \"d4297592-64e6-4022-867f-d46cb87764a7\" (UID: \"d4297592-64e6-4022-867f-d46cb87764a7\") "
Jul 27 21:43:06.590727 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-home" (OuterVolumeSpecName: "home") pod "d4297592-64e6-4022-867f-d46cb87764a7" (UID: "d4297592-64e6-4022-867f-d46cb87764a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:06.590887 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4297592-64e6-4022-867f-d46cb87764a7" (UID: "d4297592-64e6-4022-867f-d46cb87764a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:06.590946 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4297592-64e6-4022-867f-d46cb87764a7" (UID: "d4297592-64e6-4022-867f-d46cb87764a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:06.591016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590962    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-tmp-dir\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:06.591016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.590979    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-home\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:06.592955 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.592920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4297592-64e6-4022-867f-d46cb87764a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4297592-64e6-4022-867f-d46cb87764a7" (UID: "d4297592-64e6-4022-867f-d46cb87764a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:43:06.593263 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.593244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-dshm" (OuterVolumeSpecName: "dshm") pod "d4297592-64e6-4022-867f-d46cb87764a7" (UID: "d4297592-64e6-4022-867f-d46cb87764a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:06.593339 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.593307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4297592-64e6-4022-867f-d46cb87764a7-kube-api-access-zjwzm" (OuterVolumeSpecName: "kube-api-access-zjwzm") pod "d4297592-64e6-4022-867f-d46cb87764a7" (UID: "d4297592-64e6-4022-867f-d46cb87764a7"). InnerVolumeSpecName "kube-api-access-zjwzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:43:06.691843 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.691815    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-dshm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:06.691843 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.691841    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjwzm\" (UniqueName: \"kubernetes.io/projected/d4297592-64e6-4022-867f-d46cb87764a7-kube-api-access-zjwzm\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:06.691843 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.691851    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4297592-64e6-4022-867f-d46cb87764a7-tls-certs\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:06.692078 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.691860    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4297592-64e6-4022-867f-d46cb87764a7-model-cache\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:06.899240 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.899119    2577 generic.go:356] "Generic (PLEG): container finished" podID="d4297592-64e6-4022-867f-d46cb87764a7" containerID="9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692" exitCode=0
Jul 27 21:43:06.899240 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.899168    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"
Jul 27 21:43:06.899487 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.899227    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl" event={"ID":"d4297592-64e6-4022-867f-d46cb87764a7","Type":"ContainerDied","Data":"9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692"}
Jul 27 21:43:06.899487 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.899283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl" event={"ID":"d4297592-64e6-4022-867f-d46cb87764a7","Type":"ContainerDied","Data":"da1beb5a3574cb9a8da8bb0e7d45dbac506904808a0361777e8bae8f1ee29639"}
Jul 27 21:43:06.899487 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.899304    2577 scope.go:117] "RemoveContainer" containerID="9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692"
Jul 27 21:43:06.908855 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.908836    2577 scope.go:117] "RemoveContainer" containerID="9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692"
Jul 27 21:43:06.909113 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:43:06.909096    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692\": container with ID starting with 9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692 not found: ID does not exist" containerID="9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692"
Jul 27 21:43:06.909173 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.909123    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692"} err="failed to get container status \"9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692\": rpc error: code = NotFound desc = could not find container \"9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692\": container with ID starting with 9ca70b623c46fa4e1af62ab7cbd4fb2c924ee1daaa5e70096dc0bbe506bab692 not found: ID does not exist"
Jul 27 21:43:06.920570 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.920548    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"]
Jul 27 21:43:06.926326 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:06.926306    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-5d8041/ipp-route-5d8041-kserve-77bd598d67-sjsxl"]
Jul 27 21:43:07.185016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:07.184941    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4297592-64e6-4022-867f-d46cb87764a7" path="/var/lib/kubelet/pods/d4297592-64e6-4022-867f-d46cb87764a7/volumes"
Jul 27 21:43:28.341949 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:28.341911    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:29.126195 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:29.126141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:35.453692 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:35.453651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:35.529661 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:35.529626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:35.737703 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:35.737671    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:35.951149 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:35.951117    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:46.123834 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.123752    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"]
Jul 27 21:43:46.124294 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.124011    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn" podUID="58a1057c-36be-4dc7-8411-dc75dd416e3f" containerName="maas-api" containerID="cri-o://90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116" gracePeriod=30
Jul 27 21:43:46.154930 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.154901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:46.196388 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.195994    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"]
Jul 27 21:43:46.196388 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.196216    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv" podUID="15abb7d6-a85e-4017-9de4-54ca20fdfe64" containerName="payload-pre-processing" containerID="cri-o://79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc" gracePeriod=30
Jul 27 21:43:46.255493 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.255466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:46.355244 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.355214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:46.478804 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.478759    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:43:46.483100 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.483075    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:43:46.551155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.551126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a1057c-36be-4dc7-8411-dc75dd416e3f-maas-api-tls\") pod \"58a1057c-36be-4dc7-8411-dc75dd416e3f\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") "
Jul 27 21:43:46.551359 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.551163    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvzr2\" (UniqueName: \"kubernetes.io/projected/58a1057c-36be-4dc7-8411-dc75dd416e3f-kube-api-access-nvzr2\") pod \"58a1057c-36be-4dc7-8411-dc75dd416e3f\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") "
Jul 27 21:43:46.551359 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.551288    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72lfj\" (UniqueName: \"kubernetes.io/projected/15abb7d6-a85e-4017-9de4-54ca20fdfe64-kube-api-access-72lfj\") pod \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\" (UID: \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\") "
Jul 27 21:43:46.551460 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.551398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a1057c-36be-4dc7-8411-dc75dd416e3f-openshift-service-ca\") pod \"58a1057c-36be-4dc7-8411-dc75dd416e3f\" (UID: \"58a1057c-36be-4dc7-8411-dc75dd416e3f\") "
Jul 27 21:43:46.551460 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.551430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15abb7d6-a85e-4017-9de4-54ca20fdfe64-plugins-config-volume\") pod \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\" (UID: \"15abb7d6-a85e-4017-9de4-54ca20fdfe64\") "
Jul 27 21:43:46.551754 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.551731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a1057c-36be-4dc7-8411-dc75dd416e3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "58a1057c-36be-4dc7-8411-dc75dd416e3f" (UID: "58a1057c-36be-4dc7-8411-dc75dd416e3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:43:46.551987 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.551871    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15abb7d6-a85e-4017-9de4-54ca20fdfe64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15abb7d6-a85e-4017-9de4-54ca20fdfe64" (UID: "15abb7d6-a85e-4017-9de4-54ca20fdfe64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:43:46.553507 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.553477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a1057c-36be-4dc7-8411-dc75dd416e3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "58a1057c-36be-4dc7-8411-dc75dd416e3f" (UID: "58a1057c-36be-4dc7-8411-dc75dd416e3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:43:46.553609 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.553571    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a1057c-36be-4dc7-8411-dc75dd416e3f-kube-api-access-nvzr2" (OuterVolumeSpecName: "kube-api-access-nvzr2") pod "58a1057c-36be-4dc7-8411-dc75dd416e3f" (UID: "58a1057c-36be-4dc7-8411-dc75dd416e3f"). InnerVolumeSpecName "kube-api-access-nvzr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:43:46.553672 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.553645    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15abb7d6-a85e-4017-9de4-54ca20fdfe64-kube-api-access-72lfj" (OuterVolumeSpecName: "kube-api-access-72lfj") pod "15abb7d6-a85e-4017-9de4-54ca20fdfe64" (UID: "15abb7d6-a85e-4017-9de4-54ca20fdfe64"). InnerVolumeSpecName "kube-api-access-72lfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:43:46.652654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.652567    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a1057c-36be-4dc7-8411-dc75dd416e3f-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:46.652654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.652598    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvzr2\" (UniqueName: \"kubernetes.io/projected/58a1057c-36be-4dc7-8411-dc75dd416e3f-kube-api-access-nvzr2\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:46.652654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.652608    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72lfj\" (UniqueName: \"kubernetes.io/projected/15abb7d6-a85e-4017-9de4-54ca20fdfe64-kube-api-access-72lfj\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:46.652654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.652617    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a1057c-36be-4dc7-8411-dc75dd416e3f-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:46.652654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:46.652627    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15abb7d6-a85e-4017-9de4-54ca20fdfe64-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:43:47.097292 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.097256    2577 generic.go:356] "Generic (PLEG): container finished" podID="58a1057c-36be-4dc7-8411-dc75dd416e3f" containerID="90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116" exitCode=0
Jul 27 21:43:47.097454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.097306    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"
Jul 27 21:43:47.097454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.097343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn" event={"ID":"58a1057c-36be-4dc7-8411-dc75dd416e3f","Type":"ContainerDied","Data":"90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116"}
Jul 27 21:43:47.097454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.097388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn" event={"ID":"58a1057c-36be-4dc7-8411-dc75dd416e3f","Type":"ContainerDied","Data":"a08d021344c45f6b9825cc2b28b4078a88c85b8565c4defa8ee7ed5fc9ace83f"}
Jul 27 21:43:47.097454 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.097409    2577 scope.go:117] "RemoveContainer" containerID="90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116"
Jul 27 21:43:47.098909 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.098884    2577 generic.go:356] "Generic (PLEG): container finished" podID="15abb7d6-a85e-4017-9de4-54ca20fdfe64" containerID="79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc" exitCode=0
Jul 27 21:43:47.099016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.098942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv" event={"ID":"15abb7d6-a85e-4017-9de4-54ca20fdfe64","Type":"ContainerDied","Data":"79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc"}
Jul 27 21:43:47.099016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.098979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv" event={"ID":"15abb7d6-a85e-4017-9de4-54ca20fdfe64","Type":"ContainerDied","Data":"1611a0d66fd614b96d06752703431455922658ef993b0ec8d1d92b5170302bf8"}
Jul 27 21:43:47.099160 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.099143    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"
Jul 27 21:43:47.111056 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.110884    2577 scope.go:117] "RemoveContainer" containerID="90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116"
Jul 27 21:43:47.111384 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:43:47.111359    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116\": container with ID starting with 90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116 not found: ID does not exist" containerID="90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116"
Jul 27 21:43:47.111545 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.111395    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116"} err="failed to get container status \"90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116\": rpc error: code = NotFound desc = could not find container \"90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116\": container with ID starting with 90521df86ace49133b9504fcad0340a5d55f6a0a28cde25305771ed2d348c116 not found: ID does not exist"
Jul 27 21:43:47.111545 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.111419    2577 scope.go:117] "RemoveContainer" containerID="79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc"
Jul 27 21:43:47.120490 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.120467    2577 scope.go:117] "RemoveContainer" containerID="79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc"
Jul 27 21:43:47.120817 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:43:47.120772    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc\": container with ID starting with 79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc not found: ID does not exist" containerID="79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc"
Jul 27 21:43:47.120922 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.120825    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc"} err="failed to get container status \"79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc\": rpc error: code = NotFound desc = could not find container \"79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc\": container with ID starting with 79e0177eaf0b2e66681bf9d3b7b39e9bfc53c09b86787d0a6bd8007446f161cc not found: ID does not exist"
Jul 27 21:43:47.126271 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.126243    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"]
Jul 27 21:43:47.129301 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.129279    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0e6399-b4ff99cb8-vstgv"]
Jul 27 21:43:47.145730 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.145704    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"]
Jul 27 21:43:47.151858 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.151837    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0e6399-848ffcc596-txscn"]
Jul 27 21:43:47.185514 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.185478    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15abb7d6-a85e-4017-9de4-54ca20fdfe64" path="/var/lib/kubelet/pods/15abb7d6-a85e-4017-9de4-54ca20fdfe64/volumes"
Jul 27 21:43:47.185910 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:47.185892    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a1057c-36be-4dc7-8411-dc75dd416e3f" path="/var/lib/kubelet/pods/58a1057c-36be-4dc7-8411-dc75dd416e3f/volumes"
Jul 27 21:43:49.653398 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:49.653358    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:43:51.227765 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:51.227730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:51.598769 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:51.598739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:52.387100 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:52.387061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:57.823379 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:57.823307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:57.922534 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:57.922494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:58.103060 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:58.102973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:43:58.311033 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:43:58.311001    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:09.409758 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.409720    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"]
Jul 27 21:44:09.410262 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.409995    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2" podUID="d54d20a9-a111-4653-bbb9-ea5cab26bf77" containerName="maas-api" containerID="cri-o://b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25" gracePeriod=30
Jul 27 21:44:09.440086 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.440055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:09.464227 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.464171    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"]
Jul 27 21:44:09.464458 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.464429    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7" podUID="4c8ca59b-b54e-43b4-84d0-632257bf9118" containerName="payload-processing" containerID="cri-o://4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77" gracePeriod=30
Jul 27 21:44:09.474210 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.474166    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"]
Jul 27 21:44:09.474403 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.474383    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt" podUID="f9f559ef-6971-489d-9076-c5a19212336f" containerName="payload-pre-processing" containerID="cri-o://68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d" gracePeriod=30
Jul 27 21:44:09.509195 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.509155    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:09.571608 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.571490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:09.750951 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.750928    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:44:09.754719 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.754701    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:44:09.760771 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.760754    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:44:09.860321 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54d20a9-a111-4653-bbb9-ea5cab26bf77-openshift-service-ca\") pod \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") "
Jul 27 21:44:09.860321 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860330    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8ca59b-b54e-43b4-84d0-632257bf9118-plugins-config-volume\") pod \"4c8ca59b-b54e-43b4-84d0-632257bf9118\" (UID: \"4c8ca59b-b54e-43b4-84d0-632257bf9118\") "
Jul 27 21:44:09.860592 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860350    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls\") pod \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") "
Jul 27 21:44:09.860592 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq8dj\" (UniqueName: \"kubernetes.io/projected/d54d20a9-a111-4653-bbb9-ea5cab26bf77-kube-api-access-hq8dj\") pod \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\" (UID: \"d54d20a9-a111-4653-bbb9-ea5cab26bf77\") "
Jul 27 21:44:09.860592 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5pxv\" (UniqueName: \"kubernetes.io/projected/4c8ca59b-b54e-43b4-84d0-632257bf9118-kube-api-access-q5pxv\") pod \"4c8ca59b-b54e-43b4-84d0-632257bf9118\" (UID: \"4c8ca59b-b54e-43b4-84d0-632257bf9118\") "
Jul 27 21:44:09.860592 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860459    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f559ef-6971-489d-9076-c5a19212336f-plugins-config-volume\") pod \"f9f559ef-6971-489d-9076-c5a19212336f\" (UID: \"f9f559ef-6971-489d-9076-c5a19212336f\") "
Jul 27 21:44:09.860592 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz6jw\" (UniqueName: \"kubernetes.io/projected/f9f559ef-6971-489d-9076-c5a19212336f-kube-api-access-vz6jw\") pod \"f9f559ef-6971-489d-9076-c5a19212336f\" (UID: \"f9f559ef-6971-489d-9076-c5a19212336f\") "
Jul 27 21:44:09.860834 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860750    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54d20a9-a111-4653-bbb9-ea5cab26bf77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d54d20a9-a111-4653-bbb9-ea5cab26bf77" (UID: "d54d20a9-a111-4653-bbb9-ea5cab26bf77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:09.860834 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860769    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8ca59b-b54e-43b4-84d0-632257bf9118-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c8ca59b-b54e-43b4-84d0-632257bf9118" (UID: "4c8ca59b-b54e-43b4-84d0-632257bf9118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:09.860925 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860869    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f559ef-6971-489d-9076-c5a19212336f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9f559ef-6971-489d-9076-c5a19212336f" (UID: "f9f559ef-6971-489d-9076-c5a19212336f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:09.860925 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860883    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54d20a9-a111-4653-bbb9-ea5cab26bf77-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:09.860925 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.860904    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c8ca59b-b54e-43b4-84d0-632257bf9118-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:09.862857 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.862832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54d20a9-a111-4653-bbb9-ea5cab26bf77-kube-api-access-hq8dj" (OuterVolumeSpecName: "kube-api-access-hq8dj") pod "d54d20a9-a111-4653-bbb9-ea5cab26bf77" (UID: "d54d20a9-a111-4653-bbb9-ea5cab26bf77"). InnerVolumeSpecName "kube-api-access-hq8dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:09.862993 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.862878    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8ca59b-b54e-43b4-84d0-632257bf9118-kube-api-access-q5pxv" (OuterVolumeSpecName: "kube-api-access-q5pxv") pod "4c8ca59b-b54e-43b4-84d0-632257bf9118" (UID: "4c8ca59b-b54e-43b4-84d0-632257bf9118"). InnerVolumeSpecName "kube-api-access-q5pxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:09.863093 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.863069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f559ef-6971-489d-9076-c5a19212336f-kube-api-access-vz6jw" (OuterVolumeSpecName: "kube-api-access-vz6jw") pod "f9f559ef-6971-489d-9076-c5a19212336f" (UID: "f9f559ef-6971-489d-9076-c5a19212336f"). InnerVolumeSpecName "kube-api-access-vz6jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:09.863149 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.863138    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d54d20a9-a111-4653-bbb9-ea5cab26bf77" (UID: "d54d20a9-a111-4653-bbb9-ea5cab26bf77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:44:09.962216 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.962089    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d54d20a9-a111-4653-bbb9-ea5cab26bf77-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:09.962216 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.962123    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq8dj\" (UniqueName: \"kubernetes.io/projected/d54d20a9-a111-4653-bbb9-ea5cab26bf77-kube-api-access-hq8dj\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:09.962216 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.962137    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5pxv\" (UniqueName: \"kubernetes.io/projected/4c8ca59b-b54e-43b4-84d0-632257bf9118-kube-api-access-q5pxv\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:09.962216 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.962149    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f559ef-6971-489d-9076-c5a19212336f-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:09.962216 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:09.962161    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz6jw\" (UniqueName: \"kubernetes.io/projected/f9f559ef-6971-489d-9076-c5a19212336f-kube-api-access-vz6jw\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:10.211971 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.211932    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c8ca59b-b54e-43b4-84d0-632257bf9118" containerID="4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77" exitCode=0
Jul 27 21:44:10.212155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.211977    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"
Jul 27 21:44:10.212155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.212030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7" event={"ID":"4c8ca59b-b54e-43b4-84d0-632257bf9118","Type":"ContainerDied","Data":"4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77"}
Jul 27 21:44:10.212155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.212073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7" event={"ID":"4c8ca59b-b54e-43b4-84d0-632257bf9118","Type":"ContainerDied","Data":"32228490b84da021db75578bf1a77a2b2da5e195d5576086bdd1ab4fada53a66"}
Jul 27 21:44:10.212155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.212098    2577 scope.go:117] "RemoveContainer" containerID="4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77"
Jul 27 21:44:10.220477 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.220451    2577 generic.go:356] "Generic (PLEG): container finished" podID="d54d20a9-a111-4653-bbb9-ea5cab26bf77" containerID="b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25" exitCode=0
Jul 27 21:44:10.220714 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.220688    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2" event={"ID":"d54d20a9-a111-4653-bbb9-ea5cab26bf77","Type":"ContainerDied","Data":"b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25"}
Jul 27 21:44:10.220849 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.220768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2" event={"ID":"d54d20a9-a111-4653-bbb9-ea5cab26bf77","Type":"ContainerDied","Data":"c042e5033c93c867fd303e906faf57dc46684e034a452bb11712f1b683279638"}
Jul 27 21:44:10.220849 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.220691    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"
Jul 27 21:44:10.222243 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.222218    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9f559ef-6971-489d-9076-c5a19212336f" containerID="68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d" exitCode=0
Jul 27 21:44:10.222432 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.222250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt" event={"ID":"f9f559ef-6971-489d-9076-c5a19212336f","Type":"ContainerDied","Data":"68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d"}
Jul 27 21:44:10.222432 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.222271    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"
Jul 27 21:44:10.222432 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.222324    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt" event={"ID":"f9f559ef-6971-489d-9076-c5a19212336f","Type":"ContainerDied","Data":"2299fd156649a818989c60fe8246640e8f4dc90451f7b42465caff72dbd99106"}
Jul 27 21:44:10.231016 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.230996    2577 scope.go:117] "RemoveContainer" containerID="4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77"
Jul 27 21:44:10.231382 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:44:10.231319    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77\": container with ID starting with 4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77 not found: ID does not exist" containerID="4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77"
Jul 27 21:44:10.231382 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.231358    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77"} err="failed to get container status \"4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77\": rpc error: code = NotFound desc = could not find container \"4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77\": container with ID starting with 4bd1416d5130bb2aafb9513edf8cbc305f8cf5c14e919d84b97bbf7179a27c77 not found: ID does not exist"
Jul 27 21:44:10.231382 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.231375    2577 scope.go:117] "RemoveContainer" containerID="b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25"
Jul 27 21:44:10.239911 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.239889    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"]
Jul 27 21:44:10.240398 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.240371    2577 scope.go:117] "RemoveContainer" containerID="b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25"
Jul 27 21:44:10.240785 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:44:10.240733    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25\": container with ID starting with b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25 not found: ID does not exist" containerID="b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25"
Jul 27 21:44:10.240785 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.240766    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25"} err="failed to get container status \"b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25\": rpc error: code = NotFound desc = could not find container \"b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25\": container with ID starting with b1e86624ea70fabd0bdb647c087c8f65beb3fa936248461bf65a39032d2d9b25 not found: ID does not exist"
Jul 27 21:44:10.240917 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.240790    2577 scope.go:117] "RemoveContainer" containerID="68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d"
Jul 27 21:44:10.242824 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.242807    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-c910ce-6f9b65b44b-68dx7"]
Jul 27 21:44:10.249068 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.249040    2577 scope.go:117] "RemoveContainer" containerID="68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d"
Jul 27 21:44:10.249364 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:44:10.249347    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d\": container with ID starting with 68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d not found: ID does not exist" containerID="68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d"
Jul 27 21:44:10.249414 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.249372    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d"} err="failed to get container status \"68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d\": rpc error: code = NotFound desc = could not find container \"68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d\": container with ID starting with 68b74a0f574c884cd298c7fddc8575c45705b369100a57059c095665d5cc093d not found: ID does not exist"
Jul 27 21:44:10.255415 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.255393    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"]
Jul 27 21:44:10.259326 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.259305    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c910ce-5d45784bf6-npqpt"]
Jul 27 21:44:10.272572 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.272550    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"]
Jul 27 21:44:10.274428 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:10.274409    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c910ce-7d955ccfcb-vrdh2"]
Jul 27 21:44:11.186071 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:11.186040    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c8ca59b-b54e-43b4-84d0-632257bf9118" path="/var/lib/kubelet/pods/4c8ca59b-b54e-43b4-84d0-632257bf9118/volumes"
Jul 27 21:44:11.186485 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:11.186430    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54d20a9-a111-4653-bbb9-ea5cab26bf77" path="/var/lib/kubelet/pods/d54d20a9-a111-4653-bbb9-ea5cab26bf77/volumes"
Jul 27 21:44:11.186730 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:11.186717    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9f559ef-6971-489d-9076-c5a19212336f" path="/var/lib/kubelet/pods/f9f559ef-6971-489d-9076-c5a19212336f/volumes"
Jul 27 21:44:12.866849 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:12.866816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:44:13.587512 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:13.587474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:13.967256 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:13.967083    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:20.074069 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.074029    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:20.148587 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.148549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:20.354247 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.354138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:20.560908 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.560872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:20.663044 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.662960    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"]
Jul 27 21:44:20.663383 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663369    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66cf0f51-5be7-4818-acbc-4877db8e9604" containerName="maas-api"
Jul 27 21:44:20.663440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663385    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="66cf0f51-5be7-4818-acbc-4877db8e9604" containerName="maas-api"
Jul 27 21:44:20.663440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663399    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15abb7d6-a85e-4017-9de4-54ca20fdfe64" containerName="payload-pre-processing"
Jul 27 21:44:20.663440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663406    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="15abb7d6-a85e-4017-9de4-54ca20fdfe64" containerName="payload-pre-processing"
Jul 27 21:44:20.663440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663414    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50824431-23ee-4d62-876f-049811b78db9" containerName="payload-pre-processing"
Jul 27 21:44:20.663440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663419    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="50824431-23ee-4d62-876f-049811b78db9" containerName="payload-pre-processing"
Jul 27 21:44:20.663440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663429    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8ca59b-b54e-43b4-84d0-632257bf9118" containerName="payload-processing"
Jul 27 21:44:20.663440 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663434    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8ca59b-b54e-43b4-84d0-632257bf9118" containerName="payload-processing"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663443    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54d20a9-a111-4653-bbb9-ea5cab26bf77" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663449    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54d20a9-a111-4653-bbb9-ea5cab26bf77" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663457    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4297592-64e6-4022-867f-d46cb87764a7" containerName="main"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663462    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4297592-64e6-4022-867f-d46cb87764a7" containerName="main"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663469    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ebc46c9-bd63-4099-9047-2230f599af74" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663474    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ebc46c9-bd63-4099-9047-2230f599af74" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663479    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f559ef-6971-489d-9076-c5a19212336f" containerName="payload-pre-processing"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663484    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f559ef-6971-489d-9076-c5a19212336f" containerName="payload-pre-processing"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663489    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a1057c-36be-4dc7-8411-dc75dd416e3f" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663494    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a1057c-36be-4dc7-8411-dc75dd416e3f" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663553    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="66cf0f51-5be7-4818-acbc-4877db8e9604" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663561    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ebc46c9-bd63-4099-9047-2230f599af74" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663567    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8ca59b-b54e-43b4-84d0-632257bf9118" containerName="payload-processing"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663574    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54d20a9-a111-4653-bbb9-ea5cab26bf77" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663581    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a1057c-36be-4dc7-8411-dc75dd416e3f" containerName="maas-api"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663585    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="50824431-23ee-4d62-876f-049811b78db9" containerName="payload-pre-processing"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663591    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4297592-64e6-4022-867f-d46cb87764a7" containerName="main"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663596    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f559ef-6971-489d-9076-c5a19212336f" containerName="payload-pre-processing"
Jul 27 21:44:20.663658 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.663600    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="15abb7d6-a85e-4017-9de4-54ca20fdfe64" containerName="payload-pre-processing"
Jul 27 21:44:20.668287 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.668270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"
Jul 27 21:44:20.670829 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.670809    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4fvvt\""
Jul 27 21:44:20.674315 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.674291    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"]
Jul 27 21:44:20.755968 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.755929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dcc58a-a2d7-497c-b550-94876b1b1d5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8\" (UID: \"b4dcc58a-a2d7-497c-b550-94876b1b1d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"
Jul 27 21:44:20.856610 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.856566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dcc58a-a2d7-497c-b550-94876b1b1d5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8\" (UID: \"b4dcc58a-a2d7-497c-b550-94876b1b1d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"
Jul 27 21:44:20.857226 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.857176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dcc58a-a2d7-497c-b550-94876b1b1d5f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8\" (UID: \"b4dcc58a-a2d7-497c-b550-94876b1b1d5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"
Jul 27 21:44:20.980361 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:20.980334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"
Jul 27 21:44:21.116006 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:21.115978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"]
Jul 27 21:44:21.118113 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:44:21.118084    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dcc58a_a2d7_497c_b550_94876b1b1d5f.slice/crio-9a359167fade1a4f7844f1d1c2b24ae8745a32809d56e0fa45f70b70c8ec0115 WatchSource:0}: Error finding container 9a359167fade1a4f7844f1d1c2b24ae8745a32809d56e0fa45f70b70c8ec0115: Status 404 returned error can't find the container with id 9a359167fade1a4f7844f1d1c2b24ae8745a32809d56e0fa45f70b70c8ec0115
Jul 27 21:44:21.279007 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:21.278980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8" event={"ID":"b4dcc58a-a2d7-497c-b550-94876b1b1d5f","Type":"ContainerStarted","Data":"9a359167fade1a4f7844f1d1c2b24ae8745a32809d56e0fa45f70b70c8ec0115"}
Jul 27 21:44:22.285117 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:22.285077    2577 generic.go:356] "Generic (PLEG): container finished" podID="b4dcc58a-a2d7-497c-b550-94876b1b1d5f" containerID="2b6fe52ef41fa02a4ccd3d199fe6f57d92eb4da0a2edca8a6c41a55d3a1a124b" exitCode=0
Jul 27 21:44:22.285537 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:22.285128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8" event={"ID":"b4dcc58a-a2d7-497c-b550-94876b1b1d5f","Type":"ContainerDied","Data":"2b6fe52ef41fa02a4ccd3d199fe6f57d92eb4da0a2edca8a6c41a55d3a1a124b"}
Jul 27 21:44:23.423787 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:23.423762    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"
Jul 27 21:44:23.479296 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:23.479258    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dcc58a-a2d7-497c-b550-94876b1b1d5f-maas-api-service-ca\") pod \"b4dcc58a-a2d7-497c-b550-94876b1b1d5f\" (UID: \"b4dcc58a-a2d7-497c-b550-94876b1b1d5f\") "
Jul 27 21:44:23.479599 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:23.479578    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4dcc58a-a2d7-497c-b550-94876b1b1d5f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4dcc58a-a2d7-497c-b550-94876b1b1d5f" (UID: "b4dcc58a-a2d7-497c-b550-94876b1b1d5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:23.580858 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:23.580769    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4dcc58a-a2d7-497c-b550-94876b1b1d5f-maas-api-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:24.295388 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:24.295358    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"
Jul 27 21:44:24.295594 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:24.295358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8" event={"ID":"b4dcc58a-a2d7-497c-b550-94876b1b1d5f","Type":"ContainerDied","Data":"9a359167fade1a4f7844f1d1c2b24ae8745a32809d56e0fa45f70b70c8ec0115"}
Jul 27 21:44:24.295594 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:24.295481    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a359167fade1a4f7844f1d1c2b24ae8745a32809d56e0fa45f70b70c8ec0115"
Jul 27 21:44:31.294328 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.294285    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"]
Jul 27 21:44:31.294882 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.294529    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj" podUID="3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" containerName="maas-api" containerID="cri-o://042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf" gracePeriod=30
Jul 27 21:44:31.330112 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.330079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:31.357447 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.357410    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"]
Jul 27 21:44:31.358122 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.358068    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6" podUID="14b42c47-4336-4d19-a337-71bd80d68c67" containerName="payload-processing" containerID="cri-o://8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd" gracePeriod=30
Jul 27 21:44:31.360005 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.359982    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"]
Jul 27 21:44:31.360204 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.360161    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7" podUID="a1f077dd-924a-4915-8ae0-64a0ec409001" containerName="payload-pre-processing" containerID="cri-o://0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576" gracePeriod=30
Jul 27 21:44:31.400500 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.400455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:31.464538 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.464493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:31.636223 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.636201    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:44:31.639111 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.639093    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:44:31.645795 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.645779    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:44:31.752458 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752430    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75bjn\" (UniqueName: \"kubernetes.io/projected/14b42c47-4336-4d19-a337-71bd80d68c67-kube-api-access-75bjn\") pod \"14b42c47-4336-4d19-a337-71bd80d68c67\" (UID: \"14b42c47-4336-4d19-a337-71bd80d68c67\") "
Jul 27 21:44:31.752654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752483    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b42c47-4336-4d19-a337-71bd80d68c67-plugins-config-volume\") pod \"14b42c47-4336-4d19-a337-71bd80d68c67\" (UID: \"14b42c47-4336-4d19-a337-71bd80d68c67\") "
Jul 27 21:44:31.752654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752514    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phk95\" (UniqueName: \"kubernetes.io/projected/a1f077dd-924a-4915-8ae0-64a0ec409001-kube-api-access-phk95\") pod \"a1f077dd-924a-4915-8ae0-64a0ec409001\" (UID: \"a1f077dd-924a-4915-8ae0-64a0ec409001\") "
Jul 27 21:44:31.752654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752545    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-maas-api-tls\") pod \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") "
Jul 27 21:44:31.752654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752583    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-openshift-service-ca\") pod \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") "
Jul 27 21:44:31.752654 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752632    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztjx5\" (UniqueName: \"kubernetes.io/projected/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-kube-api-access-ztjx5\") pod \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\" (UID: \"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6\") "
Jul 27 21:44:31.752937 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752673    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f077dd-924a-4915-8ae0-64a0ec409001-plugins-config-volume\") pod \"a1f077dd-924a-4915-8ae0-64a0ec409001\" (UID: \"a1f077dd-924a-4915-8ae0-64a0ec409001\") "
Jul 27 21:44:31.753001 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752943    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b42c47-4336-4d19-a337-71bd80d68c67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14b42c47-4336-4d19-a337-71bd80d68c67" (UID: "14b42c47-4336-4d19-a337-71bd80d68c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:31.753001 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.752979    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" (UID: "3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:31.753297 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.753235    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f077dd-924a-4915-8ae0-64a0ec409001-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1f077dd-924a-4915-8ae0-64a0ec409001" (UID: "a1f077dd-924a-4915-8ae0-64a0ec409001"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:31.754714 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.754686    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b42c47-4336-4d19-a337-71bd80d68c67-kube-api-access-75bjn" (OuterVolumeSpecName: "kube-api-access-75bjn") pod "14b42c47-4336-4d19-a337-71bd80d68c67" (UID: "14b42c47-4336-4d19-a337-71bd80d68c67"). InnerVolumeSpecName "kube-api-access-75bjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:31.754810 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.754749    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" (UID: "3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:44:31.755326 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.755292    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-kube-api-access-ztjx5" (OuterVolumeSpecName: "kube-api-access-ztjx5") pod "3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" (UID: "3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6"). InnerVolumeSpecName "kube-api-access-ztjx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:31.755326 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.755299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f077dd-924a-4915-8ae0-64a0ec409001-kube-api-access-phk95" (OuterVolumeSpecName: "kube-api-access-phk95") pod "a1f077dd-924a-4915-8ae0-64a0ec409001" (UID: "a1f077dd-924a-4915-8ae0-64a0ec409001"). InnerVolumeSpecName "kube-api-access-phk95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:31.853952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.853861    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1f077dd-924a-4915-8ae0-64a0ec409001-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:31.853952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.853900    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bjn\" (UniqueName: \"kubernetes.io/projected/14b42c47-4336-4d19-a337-71bd80d68c67-kube-api-access-75bjn\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:31.853952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.853910    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b42c47-4336-4d19-a337-71bd80d68c67-plugins-config-volume\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:31.853952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.853919    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phk95\" (UniqueName: \"kubernetes.io/projected/a1f077dd-924a-4915-8ae0-64a0ec409001-kube-api-access-phk95\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:31.853952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.853929    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-maas-api-tls\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:31.853952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.853937    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-openshift-service-ca\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:31.853952 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:31.853946    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztjx5\" (UniqueName: \"kubernetes.io/projected/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6-kube-api-access-ztjx5\") on node \"ip-10-0-131-91.ec2.internal\" DevicePath \"\""
Jul 27 21:44:32.336131 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.336096    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" containerID="042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf" exitCode=0
Jul 27 21:44:32.336525 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.336159    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"
Jul 27 21:44:32.336525 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.336202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj" event={"ID":"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6","Type":"ContainerDied","Data":"042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf"}
Jul 27 21:44:32.336525 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.336252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj" event={"ID":"3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6","Type":"ContainerDied","Data":"210dfd843435f5cf3bfe5f60d59a24012bd2f0f2a361200b8c6383e8164a4d80"}
Jul 27 21:44:32.336525 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.336274    2577 scope.go:117] "RemoveContainer" containerID="042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf"
Jul 27 21:44:32.337980 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.337775    2577 generic.go:356] "Generic (PLEG): container finished" podID="14b42c47-4336-4d19-a337-71bd80d68c67" containerID="8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd" exitCode=0
Jul 27 21:44:32.337980 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.337846    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"
Jul 27 21:44:32.337980 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.337859    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6" event={"ID":"14b42c47-4336-4d19-a337-71bd80d68c67","Type":"ContainerDied","Data":"8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd"}
Jul 27 21:44:32.337980 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.337902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6" event={"ID":"14b42c47-4336-4d19-a337-71bd80d68c67","Type":"ContainerDied","Data":"21f55a6ef3f0618d1075ad037b331272020e1b0530a37548880b35c03cea4a85"}
Jul 27 21:44:32.339251 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.339226    2577 generic.go:356] "Generic (PLEG): container finished" podID="a1f077dd-924a-4915-8ae0-64a0ec409001" containerID="0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576" exitCode=0
Jul 27 21:44:32.339370 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.339264    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"
Jul 27 21:44:32.339370 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.339332    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7" event={"ID":"a1f077dd-924a-4915-8ae0-64a0ec409001","Type":"ContainerDied","Data":"0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576"}
Jul 27 21:44:32.339461 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.339376    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7" event={"ID":"a1f077dd-924a-4915-8ae0-64a0ec409001","Type":"ContainerDied","Data":"9e878d76c5576832a07e2bb07681c359f8b4e8b66ab72c75c7287dfd4f812615"}
Jul 27 21:44:32.351996 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.351978    2577 scope.go:117] "RemoveContainer" containerID="042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf"
Jul 27 21:44:32.352458 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:44:32.352406    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf\": container with ID starting with 042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf not found: ID does not exist" containerID="042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf"
Jul 27 21:44:32.352458 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.352433    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf"} err="failed to get container status \"042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf\": rpc error: code = NotFound desc = could not find container \"042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf\": container with ID starting with 042ec81b1f991dce9a4a26d92905a320aab634999443fffff9ebec157f0d1ccf not found: ID does not exist"
Jul 27 21:44:32.352458 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.352455    2577 scope.go:117] "RemoveContainer" containerID="8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd"
Jul 27 21:44:32.362923 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.362906    2577 scope.go:117] "RemoveContainer" containerID="8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd"
Jul 27 21:44:32.363155 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:44:32.363136    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd\": container with ID starting with 8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd not found: ID does not exist" containerID="8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd"
Jul 27 21:44:32.363267 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.363162    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd"} err="failed to get container status \"8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd\": rpc error: code = NotFound desc = could not find container \"8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd\": container with ID starting with 8d4baf6495ddd7561f54d3975f7356f50ffb14d42908f1edb4338ba38838cfbd not found: ID does not exist"
Jul 27 21:44:32.363267 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.363205    2577 scope.go:117] "RemoveContainer" containerID="0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576"
Jul 27 21:44:32.365343 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.365310    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"]
Jul 27 21:44:32.366838 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.366819    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-85f004-675bb5f5ff-8sdzj"]
Jul 27 21:44:32.371292 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.371266    2577 scope.go:117] "RemoveContainer" containerID="0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576"
Jul 27 21:44:32.371559 ip-10-0-131-91 kubenswrapper[2577]: E0727 21:44:32.371539    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576\": container with ID starting with 0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576 not found: ID does not exist" containerID="0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576"
Jul 27 21:44:32.371611 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.371566    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576"} err="failed to get container status \"0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576\": rpc error: code = NotFound desc = could not find container \"0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576\": container with ID starting with 0849098cbaa72423a1f46feed2b7ec585f883d38a0c6fa5434f531686b28f576 not found: ID does not exist"
Jul 27 21:44:32.376093 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.376074    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"]
Jul 27 21:44:32.380140 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.380121    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-85f004-764dddccdd-vr9x7"]
Jul 27 21:44:32.389913 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.389891    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"]
Jul 27 21:44:32.393845 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:32.393826    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-85f004-544ccb77cd-98bc6"]
Jul 27 21:44:33.186233 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:33.186202    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14b42c47-4336-4d19-a337-71bd80d68c67" path="/var/lib/kubelet/pods/14b42c47-4336-4d19-a337-71bd80d68c67/volumes"
Jul 27 21:44:33.186581 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:33.186568    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" path="/var/lib/kubelet/pods/3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6/volumes"
Jul 27 21:44:33.186869 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:33.186858    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1f077dd-924a-4915-8ae0-64a0ec409001" path="/var/lib/kubelet/pods/a1f077dd-924a-4915-8ae0-64a0ec409001/volumes"
Jul 27 21:44:34.061910 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:34.061880    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-5pnw4"]
Jul 27 21:44:35.770820 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:35.770786    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"]
Jul 27 21:44:35.775263 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:35.775238    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-85f004-qfvx8"]
Jul 27 21:44:35.776612 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:35.776588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:36.133214 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:36.133121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:37.186424 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:37.186390    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4dcc58a-a2d7-497c-b550-94876b1b1d5f" path="/var/lib/kubelet/pods/b4dcc58a-a2d7-497c-b550-94876b1b1d5f/volumes"
Jul 27 21:44:41.945234 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:41.945173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:42.043259 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:42.043225    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:42.196774 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:42.196670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:42.411734 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:42.411678    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:48.543367 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:48.543334    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc_f864b196-c1f5-446e-8a15-6881ca9876c1/copy-manifests/0.log"
Jul 27 21:44:48.550020 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:48.549996    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc_f864b196-c1f5-446e-8a15-6881ca9876c1/manager/0.log"
Jul 27 21:44:49.099598 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:49.099568    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753115-wnssw_11c1c350-9f1c-43c2-872b-b981b1bbfa92/cleanup/0.log"
Jul 27 21:44:49.195027 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:49.195001    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753130-v2szg_9ae0274c-74c9-43ec-bedb-8915a04ee87f/cleanup/0.log"
Jul 27 21:44:50.736743 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:50.736680    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8d4rc_5d005d85-8600-4beb-a199-9385529883b0/manager/0.log"
Jul 27 21:44:51.165670 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:51.165596    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-5pnw4_2e2c3270-06b7-4b2d-b6de-7d240ddc58e9/limitador/0.log"
Jul 27 21:44:51.273700 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:51.273664    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-c2s7v_9aefded3-dd1f-45a3-82d4-f508b9b777d9/manager/0.log"
Jul 27 21:44:51.921798 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:51.921768    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f4ccdc5cb-78mtn_aa7add0a-17fa-4a49-83d4-533d5cff0a6c/kube-auth-proxy/0.log"
Jul 27 21:44:53.237977 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:53.237935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-xqv49_959b6e53-1748-4811-b3a6-2b5ef87207ef/main/0.log"
Jul 27 21:44:58.578781 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.578746    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d7w9b/must-gather-89rt2"]
Jul 27 21:44:58.579155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579078    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dcc58a-a2d7-497c-b550-94876b1b1d5f" containerName="revoke-keys"
Jul 27 21:44:58.579155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579089    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dcc58a-a2d7-497c-b550-94876b1b1d5f" containerName="revoke-keys"
Jul 27 21:44:58.579155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579100    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f077dd-924a-4915-8ae0-64a0ec409001" containerName="payload-pre-processing"
Jul 27 21:44:58.579155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579106    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f077dd-924a-4915-8ae0-64a0ec409001" containerName="payload-pre-processing"
Jul 27 21:44:58.579155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579121    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b42c47-4336-4d19-a337-71bd80d68c67" containerName="payload-processing"
Jul 27 21:44:58.579155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579126    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b42c47-4336-4d19-a337-71bd80d68c67" containerName="payload-processing"
Jul 27 21:44:58.579155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579139    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" containerName="maas-api"
Jul 27 21:44:58.579155 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579145    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" containerName="maas-api"
Jul 27 21:44:58.579443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579207    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b42c47-4336-4d19-a337-71bd80d68c67" containerName="payload-processing"
Jul 27 21:44:58.579443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579218    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed43bbb-ea5d-486c-bb0c-1d7ae8b9e1d6" containerName="maas-api"
Jul 27 21:44:58.579443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579223    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dcc58a-a2d7-497c-b550-94876b1b1d5f" containerName="revoke-keys"
Jul 27 21:44:58.579443 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.579230    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1f077dd-924a-4915-8ae0-64a0ec409001" containerName="payload-pre-processing"
Jul 27 21:44:58.583529 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.583512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d7w9b/must-gather-89rt2"
Jul 27 21:44:58.587407 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.587386    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d7w9b\"/\"kube-root-ca.crt\""
Jul 27 21:44:58.587957 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.587944    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d7w9b\"/\"openshift-service-ca.crt\""
Jul 27 21:44:58.597556 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.597533    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d7w9b/must-gather-89rt2"]
Jul 27 21:44:58.629018 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.628979    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:58.684387 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.684354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs4hx\" (UniqueName: \"kubernetes.io/projected/873f793f-6eef-4b07-bb47-3d45ba9e5351-kube-api-access-gs4hx\") pod \"must-gather-89rt2\" (UID: \"873f793f-6eef-4b07-bb47-3d45ba9e5351\") " pod="openshift-must-gather-d7w9b/must-gather-89rt2"
Jul 27 21:44:58.684563 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.684402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/873f793f-6eef-4b07-bb47-3d45ba9e5351-must-gather-output\") pod \"must-gather-89rt2\" (UID: \"873f793f-6eef-4b07-bb47-3d45ba9e5351\") " pod="openshift-must-gather-d7w9b/must-gather-89rt2"
Jul 27 21:44:58.698597 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.698564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log" limit=1024
Jul 27 21:44:58.785728 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.785688    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs4hx\" (UniqueName: \"kubernetes.io/projected/873f793f-6eef-4b07-bb47-3d45ba9e5351-kube-api-access-gs4hx\") pod \"must-gather-89rt2\" (UID: \"873f793f-6eef-4b07-bb47-3d45ba9e5351\") " pod="openshift-must-gather-d7w9b/must-gather-89rt2"
Jul 27 21:44:58.785728 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.785725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/873f793f-6eef-4b07-bb47-3d45ba9e5351-must-gather-output\") pod \"must-gather-89rt2\" (UID: \"873f793f-6eef-4b07-bb47-3d45ba9e5351\") " pod="openshift-must-gather-d7w9b/must-gather-89rt2"
Jul 27 21:44:58.786066 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.786050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/873f793f-6eef-4b07-bb47-3d45ba9e5351-must-gather-output\") pod \"must-gather-89rt2\" (UID: \"873f793f-6eef-4b07-bb47-3d45ba9e5351\") " pod="openshift-must-gather-d7w9b/must-gather-89rt2"
Jul 27 21:44:58.795065 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.795040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs4hx\" (UniqueName: \"kubernetes.io/projected/873f793f-6eef-4b07-bb47-3d45ba9e5351-kube-api-access-gs4hx\") pod \"must-gather-89rt2\" (UID: \"873f793f-6eef-4b07-bb47-3d45ba9e5351\") " pod="openshift-must-gather-d7w9b/must-gather-89rt2"
Jul 27 21:44:58.892722 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:58.892616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d7w9b/must-gather-89rt2"
Jul 27 21:44:59.050899 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:59.050874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d7w9b/must-gather-89rt2"]
Jul 27 21:44:59.053311 ip-10-0-131-91 kubenswrapper[2577]: W0727 21:44:59.053281    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod873f793f_6eef_4b07_bb47_3d45ba9e5351.slice/crio-af0f9a2be858a255052fdf7dfaa38a4e380ffbb56a6c87cb496e194db9091901 WatchSource:0}: Error finding container af0f9a2be858a255052fdf7dfaa38a4e380ffbb56a6c87cb496e194db9091901: Status 404 returned error can't find the container with id af0f9a2be858a255052fdf7dfaa38a4e380ffbb56a6c87cb496e194db9091901
Jul 27 21:44:59.055037 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:59.055019    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:44:59.472123 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:44:59.472092    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d7w9b/must-gather-89rt2" event={"ID":"873f793f-6eef-4b07-bb47-3d45ba9e5351","Type":"ContainerStarted","Data":"af0f9a2be858a255052fdf7dfaa38a4e380ffbb56a6c87cb496e194db9091901"}
Jul 27 21:45:00.479940 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:00.479888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d7w9b/must-gather-89rt2" event={"ID":"873f793f-6eef-4b07-bb47-3d45ba9e5351","Type":"ContainerStarted","Data":"0e78ec86b062a65a5fb46b7f1afa3cf88fda8e808e9ca93286dc9f9e0a07ead3"}
Jul 27 21:45:00.479940 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:00.479949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d7w9b/must-gather-89rt2" event={"ID":"873f793f-6eef-4b07-bb47-3d45ba9e5351","Type":"ContainerStarted","Data":"29aad54537ba0a22fd31062da97fd2c19cbf74ad17001d298c5f5a1693926e71"}
Jul 27 21:45:00.497776 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:00.497709    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d7w9b/must-gather-89rt2" podStartSLOduration=1.7357471960000002 podStartE2EDuration="2.497689198s" totalImagesPullingTime="761.942002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:00.496783342 +0000 UTC m=+3217.862349189" watchObservedRunningTime="2026-07-27 21:45:00.497689198 +0000 UTC m=+3217.863255058"
Jul 27 21:45:01.508917 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:01.508885    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4br5p_59a571a8-5d4f-4cb9-8010-f2257b46dbd3/global-pull-secret-syncer/0.log"
Jul 27 21:45:01.672673 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:01.672643    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qqsfw_82f64a3a-a74e-42bc-ac68-9f087db03925/konnectivity-agent/0.log"
Jul 27 21:45:01.792047 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:01.791963    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-91.ec2.internal_ed929be85ebfd2f1b71d7e50ffae918d/haproxy/0.log"
Jul 27 21:45:07.470102 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:07.469895    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-8d4rc_5d005d85-8600-4beb-a199-9385529883b0/manager/0.log"
Jul 27 21:45:07.710717 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:07.710690    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-5pnw4_2e2c3270-06b7-4b2d-b6de-7d240ddc58e9/limitador/0.log"
Jul 27 21:45:07.800655 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:07.800625    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-c2s7v_9aefded3-dd1f-45a3-82d4-f508b9b777d9/manager/0.log"
Jul 27 21:45:09.471167 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.471091    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32d01aa4-5e7d-4827-9228-2ae9422e5705/alertmanager/0.log"
Jul 27 21:45:09.492591 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.492566    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32d01aa4-5e7d-4827-9228-2ae9422e5705/config-reloader/0.log"
Jul 27 21:45:09.519222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.519131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32d01aa4-5e7d-4827-9228-2ae9422e5705/kube-rbac-proxy-web/0.log"
Jul 27 21:45:09.541887 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.541857    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32d01aa4-5e7d-4827-9228-2ae9422e5705/kube-rbac-proxy/0.log"
Jul 27 21:45:09.562958 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.562930    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32d01aa4-5e7d-4827-9228-2ae9422e5705/kube-rbac-proxy-metric/0.log"
Jul 27 21:45:09.589493 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.589465    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32d01aa4-5e7d-4827-9228-2ae9422e5705/prom-label-proxy/0.log"
Jul 27 21:45:09.614387 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.614357    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32d01aa4-5e7d-4827-9228-2ae9422e5705/init-config-reloader/0.log"
Jul 27 21:45:09.952176 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.952132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqsrq_531d1f22-d6c8-4f48-9add-d4b6729f56a1/node-exporter/0.log"
Jul 27 21:45:09.970346 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.970314    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqsrq_531d1f22-d6c8-4f48-9add-d4b6729f56a1/kube-rbac-proxy/0.log"
Jul 27 21:45:09.990824 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:09.990797    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqsrq_531d1f22-d6c8-4f48-9add-d4b6729f56a1/init-textfile/0.log"
Jul 27 21:45:10.101401 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.101359    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4f016d1-a1bf-4ce3-8935-86f591640eb8/prometheus/0.log"
Jul 27 21:45:10.123345 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.123317    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4f016d1-a1bf-4ce3-8935-86f591640eb8/config-reloader/0.log"
Jul 27 21:45:10.143602 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.143569    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4f016d1-a1bf-4ce3-8935-86f591640eb8/thanos-sidecar/0.log"
Jul 27 21:45:10.161947 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.161921    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4f016d1-a1bf-4ce3-8935-86f591640eb8/kube-rbac-proxy-web/0.log"
Jul 27 21:45:10.181355 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.181330    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4f016d1-a1bf-4ce3-8935-86f591640eb8/kube-rbac-proxy/0.log"
Jul 27 21:45:10.200933 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.200907    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4f016d1-a1bf-4ce3-8935-86f591640eb8/kube-rbac-proxy-thanos/0.log"
Jul 27 21:45:10.226325 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.226252    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b4f016d1-a1bf-4ce3-8935-86f591640eb8/init-config-reloader/0.log"
Jul 27 21:45:10.251465 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.251422    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-wdzz2_1e1fc0d7-c7bb-4405-8a9c-8c62265b3641/prometheus-operator/0.log"
Jul 27 21:45:10.268689 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.268645    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-wdzz2_1e1fc0d7-c7bb-4405-8a9c-8c62265b3641/kube-rbac-proxy/0.log"
Jul 27 21:45:10.404507 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.404433    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696ff8764c-s9mtr_c9978dcf-aeb8-400b-a957-450f83fb6b9a/thanos-query/0.log"
Jul 27 21:45:10.429070 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.429041    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696ff8764c-s9mtr_c9978dcf-aeb8-400b-a957-450f83fb6b9a/kube-rbac-proxy-web/0.log"
Jul 27 21:45:10.448392 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.448365    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696ff8764c-s9mtr_c9978dcf-aeb8-400b-a957-450f83fb6b9a/kube-rbac-proxy/0.log"
Jul 27 21:45:10.472298 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.472268    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696ff8764c-s9mtr_c9978dcf-aeb8-400b-a957-450f83fb6b9a/prom-label-proxy/0.log"
Jul 27 21:45:10.492089 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.492058    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696ff8764c-s9mtr_c9978dcf-aeb8-400b-a957-450f83fb6b9a/kube-rbac-proxy-rules/0.log"
Jul 27 21:45:10.515385 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:10.515340    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-696ff8764c-s9mtr_c9978dcf-aeb8-400b-a957-450f83fb6b9a/kube-rbac-proxy-metrics/0.log"
Jul 27 21:45:11.434052 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.433996    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"]
Jul 27 21:45:11.440463 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.440432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.443233 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.443211    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d7w9b\"/\"default-dockercfg-vqvld\""
Jul 27 21:45:11.447630 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.447606    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"]
Jul 27 21:45:11.513502 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.513462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-lib-modules\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.514119 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.514092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-podres\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.514310 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.514290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-sys\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.514518 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.514501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-proc\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.514717 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.514701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8qt\" (UniqueName: \"kubernetes.io/projected/a097b324-eb44-4191-a1cb-5ef6138e0a3d-kube-api-access-mm8qt\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.615496 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.615460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-sys\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.615699 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.615536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-proc\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.615699 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.615575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8qt\" (UniqueName: \"kubernetes.io/projected/a097b324-eb44-4191-a1cb-5ef6138e0a3d-kube-api-access-mm8qt\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.615699 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.615611    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-lib-modules\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.615699 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.615673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-podres\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.615921 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.615833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-podres\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.615921 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.615900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-sys\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.616025 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.615947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-proc\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.616259 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.616238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a097b324-eb44-4191-a1cb-5ef6138e0a3d-lib-modules\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.628816 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.628788    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8qt\" (UniqueName: \"kubernetes.io/projected/a097b324-eb44-4191-a1cb-5ef6138e0a3d-kube-api-access-mm8qt\") pod \"perf-node-gather-daemonset-87pt6\" (UID: \"a097b324-eb44-4191-a1cb-5ef6138e0a3d\") " pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.760082 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.760041    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:11.955222 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:11.955158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"]
Jul 27 21:45:12.566164 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:12.566127    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6" event={"ID":"a097b324-eb44-4191-a1cb-5ef6138e0a3d","Type":"ContainerStarted","Data":"ee8c4acdb9d7af97c67ca2c6cb480def3cfc6d2ce0cea13ed1d8bef60562e0af"}
Jul 27 21:45:12.566610 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:12.566206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6" event={"ID":"a097b324-eb44-4191-a1cb-5ef6138e0a3d","Type":"ContainerStarted","Data":"72a39fdba0a556483f841598c76d4541b9e6b428f0b32889f19f0e594f65e143"}
Jul 27 21:45:12.566610 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:12.566236    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:12.582645 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:12.582589    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6" podStartSLOduration=1.58257117 podStartE2EDuration="1.58257117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:12.5802546 +0000 UTC m=+3229.945820446" watchObservedRunningTime="2026-07-27 21:45:12.58257117 +0000 UTC m=+3229.948137020"
Jul 27 21:45:14.354475 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:14.354397    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-48trx_f2f5a733-8e79-4afd-8bf1-3b8ced670805/dns/0.log"
Jul 27 21:45:14.374254 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:14.374227    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-48trx_f2f5a733-8e79-4afd-8bf1-3b8ced670805/kube-rbac-proxy/0.log"
Jul 27 21:45:14.535331 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:14.535272    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x5xkb_214751b9-5778-4f01-8602-5b89512a8384/dns-node-resolver/0.log"
Jul 27 21:45:15.157362 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:15.157327    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ph9gv_739950fc-859d-4145-b60c-b5dc0811d195/node-ca/0.log"
Jul 27 21:45:16.358768 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:16.358739    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f4ccdc5cb-78mtn_aa7add0a-17fa-4a49-83d4-533d5cff0a6c/kube-auth-proxy/0.log"
Jul 27 21:45:17.428798 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:17.428773    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hml2p_6f7262a1-1f6a-4252-a2dc-99b96a2422c2/serve-healthcheck-canary/0.log"
Jul 27 21:45:17.973642 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:17.973616    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g7ml4_54e17ea4-433d-4bac-8a82-50ca06d6c5a0/kube-rbac-proxy/0.log"
Jul 27 21:45:17.992073 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:17.992044    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g7ml4_54e17ea4-433d-4bac-8a82-50ca06d6c5a0/exporter/0.log"
Jul 27 21:45:18.012523 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:18.012459    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g7ml4_54e17ea4-433d-4bac-8a82-50ca06d6c5a0/extractor/0.log"
Jul 27 21:45:18.586047 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:18.585494    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d7w9b/perf-node-gather-daemonset-87pt6"
Jul 27 21:45:21.257434 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:21.257403    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc_f864b196-c1f5-446e-8a15-6881ca9876c1/manager/0.log"
Jul 27 21:45:21.278543 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:21.278516    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b4c7b8d87-c2ztc_f864b196-c1f5-446e-8a15-6881ca9876c1/copy-manifests/0.log"
Jul 27 21:45:22.017764 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:22.017738    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f4ccdc5cb-78mtn_aa7add0a-17fa-4a49-83d4-533d5cff0a6c/kube-auth-proxy/0.log"
Jul 27 21:45:22.582562 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:22.582477    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-58799fcf76-wv6hz_82624754-9748-48c4-ae54-1646f9c036e7/manager/0.log"
Jul 27 21:45:23.746965 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:23.746928    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f4ccdc5cb-78mtn_aa7add0a-17fa-4a49-83d4-533d5cff0a6c/kube-auth-proxy/0.log"
Jul 27 21:45:27.369699 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:27.369672    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dn44k_236b13d5-7d68-4494-8f5d-108b30e2fe9e/migrator/0.log"
Jul 27 21:45:27.386669 ip-10-0-131-91 kubenswrapper[2577]: I0727 21:45:27.386642    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dn44k_236b13d5-7d68-4494-8f5d-108b30e2fe9e/graceful-termination/0.log"