Jul 28 19:36:21.417169 ip-10-0-138-173 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 19:36:21.807499 ip-10-0-138-173 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:36:21.807499 ip-10-0-138-173 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 19:36:21.807499 ip-10-0-138-173 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:36:21.807499 ip-10-0-138-173 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 19:36:21.807499 ip-10-0-138-173 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:36:21.809303 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.809235    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 19:36:21.815245 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815222    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:36:21.815245 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815240    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:36:21.815245 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815246    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:36:21.815245 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815251    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815262    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815266    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815273    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815279    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815283    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815288    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815298    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815303    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815307    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815311    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815315    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815319    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815326    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815335    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815340    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815345    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815350    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815354    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:36:21.815480 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815359    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815365    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815371    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815377    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815381    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815386    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815390    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815399    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815404    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815408    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815415    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815419    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815424    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815429    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815434    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815440    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815445    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815450    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815455    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815464    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:36:21.816117 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815468    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815474    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815479    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815483    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815487    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815491    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815496    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815500    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815504    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815508    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815513    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815522    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815526    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815531    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815535    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815540    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815544    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815549    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815553    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815558    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:36:21.816715 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815563    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815567    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815572    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815576    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815585    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815592    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815596    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815602    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815607    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815611    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815615    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815620    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815624    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815629    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815635    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815644    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815649    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815653    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815658    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:36:21.817571 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815662    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815666    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815671    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815675    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815679    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815684    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815689    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815693    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815697    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815706    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815710    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815715    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815719    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815725    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815729    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815733    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815738    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:36:21.818384 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.815742    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818423    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818434    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818440    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818444    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818451    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818456    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818466    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818471    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818476    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818481    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818487    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818492    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818496    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818501    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818506    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818510    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818514    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818519    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818528    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:36:21.819129 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818532    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818537    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818541    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818546    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818550    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818555    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818559    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818564    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818568    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818572    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818579    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818587    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818596    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818601    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818607    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818612    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818617    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818623    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818628    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818633    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:36:21.820037 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818637    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818642    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818647    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818651    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818662    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818666    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818671    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818675    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818679    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818684    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818688    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818692    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818697    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818701    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818706    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818710    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818714    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818723    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818727    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818732    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:36:21.820787 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818736    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818740    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818745    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818750    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818754    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818759    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818764    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818769    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818774    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818782    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818786    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818791    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818795    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818800    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818804    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818809    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818813    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818818    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818822    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818827    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:36:21.821305 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818831    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818836    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818844    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818849    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818853    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818857    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818861    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818869    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818876    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818881    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818903    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818911    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818916    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818921    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818933    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818937    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818943    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818948    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818954    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:36:21.821777 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.818959    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.819985    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.819994    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.819999    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820005    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820010    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820015    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820019    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820024    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820034    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820039    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820044    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820048    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820052    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820056    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820060    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820065    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820069    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820074    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:36:21.822275 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820078    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820083    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820094    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820101    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820106    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820112    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820116    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820121    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820129    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820135    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820140    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820145    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820150    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820155    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820160    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820169    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820174    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820179    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820183    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:36:21.822743 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820188    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820192    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820196    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820201    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820205    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820210    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820215    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820219    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820228    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820233    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820237    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820242    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820246    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820250    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820255    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820259    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820264    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820268    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820272    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:36:21.823231 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820276    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820281    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820290    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820295    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820300    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820304    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820308    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820313    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820317    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820321    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820326    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820331    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820335    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820339    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820350    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820355    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820359    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820364    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820368    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820374    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:36:21.823695 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820378    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820382    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820387    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820391    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820395    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820400    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820409    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820414    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820418    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820422    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820427    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820433    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820438    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820442    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820447    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820451    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820458    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820469    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820475    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820480    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:36:21.824229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820485    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820489    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.820494    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820757    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820771    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820778    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820783    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820787    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820791    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820810    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820814    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820818    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820821    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820825    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820828    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820831    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820834    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820837    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820840    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820843    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820846    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820850    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820853    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820856    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 19:36:21.824714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820859    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820864    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820870    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820873    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820878    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820884    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820899    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820902    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820907    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820910    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820915    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820920    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820928    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820931    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820934    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820944    2555 flags.go:64] FLAG: --enable-server="true"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820947    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820952    2555 flags.go:64] FLAG: --event-burst="100"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820955    2555 flags.go:64] FLAG: --event-qps="50"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820958    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820961    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820964    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820968    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820971    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820974    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 19:36:21.825457 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820977    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820980    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820983    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820988    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820991    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820993    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820996    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.820999    2555 flags.go:64] FLAG: --feature-gates=""
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821003    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821006    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821009    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821012    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821015    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821018    2555 flags.go:64] FLAG: --help="false"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821021    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-138-173.ec2.internal"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821024    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821027    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821030    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821033    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821039    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821042    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821045    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821048    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 19:36:21.826072 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821051    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821054    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821057    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821060    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821063    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821066    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821069    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821072    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821075    2555 flags.go:64] FLAG: --lock-file=""
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821078    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821080    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821083    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821089    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821092    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821095    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821098    2555 flags.go:64] FLAG: --logging-format="text"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821100    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821104    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821106    2555 flags.go:64] FLAG: --manifest-url=""
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821109    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821114    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821117    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821121    2555 flags.go:64] FLAG: --max-pods="110"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821124    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821127    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 19:36:21.826663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821129    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821132    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821135    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821139    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821143    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821150    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821153    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821156    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821159    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821162    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821167    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821170    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821173    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821176    2555 flags.go:64] FLAG: --port="10250"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821179    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821182    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04829ae470191192e"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821185    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821188    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821191    2555 flags.go:64] FLAG: --register-node="true"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821194    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821197    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821201    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821204    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821207    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821210    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 19:36:21.827356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821213    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821219    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821222    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821225    2555 flags.go:64] FLAG: --runonce="false"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821228    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821231    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821234    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821237    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821239    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821242    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821247    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821250    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821253    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821256    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821259    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821262    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821265    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821268    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821270    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821276    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821279    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821282    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821286    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821289    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821291    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 19:36:21.828020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821294    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821297    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821300    2555 flags.go:64] FLAG: --v="2"
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821304    2555 flags.go:64] FLAG: --version="false"
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821308    2555 flags.go:64] FLAG: --vmodule=""
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821312    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.821316    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821428    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821434    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821437    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821440    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821443    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821446    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821449    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821452    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821456    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821460    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821464    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821467    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821470    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821473    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:36:21.828607 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821476    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821479    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821481    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821484    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821486    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821489    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821491    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821494    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821496    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821499    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821502    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821504    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821507    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821509    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821512    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821514    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821518    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821520    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821523    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821527    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:36:21.829189 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821530    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821532    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821535    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821537    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821540    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821542    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821545    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821548    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821550    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821553    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821555    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821558    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821561    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821563    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821566    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821568    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821571    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821574    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821576    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821579    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:36:21.829668 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821581    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821584    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821587    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821589    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821592    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821594    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821597    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821599    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821602    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821605    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821607    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821611    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821614    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821617    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821619    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821622    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821624    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821628    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821632    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821635    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:36:21.830168 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821638    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821641    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821644    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821646    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821649    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821652    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821654    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821657    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821659    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821662    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821664    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821667    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821669    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821672    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821674    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821677    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821679    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821683    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821687    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:36:21.830649 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821690    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821693    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821695    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821699    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821703    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.821706    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.822394    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.828407    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.828422    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.829192    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.829201    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829255    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829260    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829263    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829266    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:36:21.831142 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829269    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829272    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829275    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829277    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829280    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829284    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829288    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829291    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829293    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829297    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829299    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829302    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829305    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829307    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829310    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829313    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829316    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829318    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829321    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829323    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:36:21.831511 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829326    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829328    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829331    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829333    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829336    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829340    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829343    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829345    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829348    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829351    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829353    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829356    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829358    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829361    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829363    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829366    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829368    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829371    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829373    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829376    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:36:21.832023 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829378    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829381    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829383    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829386    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829389    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829391    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829394    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829396    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829399    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829401    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829404    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829406    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829409    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829412    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829415    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829417    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829420    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829423    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829428    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:36:21.832542 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829431    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829434    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829437    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829440    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829442    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829445    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829447    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829450    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829453    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829455    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829458    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829460    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829463    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829465    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829468    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829471    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829474    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829476    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829479    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829482    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:36:21.833038 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829484    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829487    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829489    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829492    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829495    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829497    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829500    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829503    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829507    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829510    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829513    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829515    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829518    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829521    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829523    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:36:21.833527 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829526    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.829531    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829649    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829654    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829657    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829660    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829663    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829665    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829668    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829671    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829674    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829677    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829680    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829682    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829685    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:36:21.833900 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829688    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829690    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829693    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829696    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829700    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829705    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829708    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829711    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829714    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829717    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829720    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829722    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829726    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829729    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829731    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829735    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829737    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829740    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829742    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:36:21.834281 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829745    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829747    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829750    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829752    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829755    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829757    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829760    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829762    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829765    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829767    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829769    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829772    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829774    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829777    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829779    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829781    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829784    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829787    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829789    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:36:21.834745 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829791    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829794    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829797    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829799    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829802    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829804    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829807    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829809    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829811    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829814    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829818    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829820    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829824    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829827    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829830    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829833    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829836    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829839    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829841    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829844    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:36:21.835267 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829847    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829849    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829852    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829854    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829857    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829859    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829862    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829864    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829867    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829869    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829871    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829874    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829876    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829879    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829882    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829898    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829903    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829906    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829908    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829911    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:36:21.835753 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829914    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829916    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829919    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829924    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829926    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829929    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829934    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:21.829936    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.829941    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.830044    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.831974    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.833433    2555 server.go:1019] "Starting client certificate rotation"
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.833530    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 19:36:21.836289 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.833557    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 19:36:21.836617 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.836039    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 19:36:21.837371 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.837357    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 19:36:21.851443 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.851427    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 28 19:36:21.854310 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.854296    2555 log.go:25] "Validated CRI v1 image API"
Jul 28 19:36:21.855010 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.854997    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 19:36:21.857581 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.857522    2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bd151fb3-755b-49ee-abd2-f084505cec35:/dev/nvme0n1p4 eca9bc13-0f0c-4a60-b1f9-5ee653e0732d:/dev/nvme0n1p3]
Jul 28 19:36:21.857722 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.857696    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 19:36:21.862601 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.862462    2555 manager.go:216] Machine: {Timestamp:2026-07-28 19:36:21.861221485 +0000 UTC m=+0.321980076 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099323 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec245e0d023bd147ae3de8e28eb41bdd SystemUUID:ec245e0d-023b-d147-ae3d-e8e28eb41bdd BootID:b617255f-8a0b-4ec2-afe7-bf18a7944a68 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2e:02:ea:60:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2e:02:ea:60:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:0e:86:0e:f4:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 19:36:21.862601 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.862584    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 19:36:21.862741 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.862669    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 19:36:21.862741 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.862672    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 19:36:21.865517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.865497    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 19:36:21.865643 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.865520    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 19:36:21.866098 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.866085    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 19:36:21.866135 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.866100    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 19:36:21.866135 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.866107    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 19:36:21.866782 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.866772    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 19:36:21.867897 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.867879    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 19:36:21.871763 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.871751    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 19:36:21.871860 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.871851    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 19:36:21.871917 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.871878    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 19:36:21.871917 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.871901    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 19:36:21.871917 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.871917    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 19:36:21.872048 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.871926    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 28 19:36:21.872048 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.871934    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 19:36:21.872242 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.872230    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 19:36:21.872285 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.872249    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 19:36:21.874431 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.874415    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 19:36:21.875827 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.875814    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 19:36:21.875870 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.875844    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 19:36:21.876437 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876426    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 19:36:21.876468 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876443    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 19:36:21.876468 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876450    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 19:36:21.876468 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876455    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 19:36:21.876468 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876464    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 19:36:21.876468 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876469    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 19:36:21.876590 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876517    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 19:36:21.876590 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876524    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 19:36:21.876590 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876530    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 19:36:21.876590 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876535    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 19:36:21.876590 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876544    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 19:36:21.876590 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.876551    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 19:36:21.877501 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.877490    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 19:36:21.877501 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.877502    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 19:36:21.880016 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.879990    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 19:36:21.880119 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.880062    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 19:36:21.880119 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.880093    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 19:36:21.880717 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.880705    2555 server.go:1268] "Started kubelet"
Jul 28 19:36:21.880794 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.880770    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 19:36:21.880920 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.880875    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 19:36:21.880990 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.880925    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 19:36:21.881746 ip-10-0-138-173 systemd[1]: Started Kubernetes Kubelet.
Jul 28 19:36:21.882496 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.882350    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 19:36:21.882920 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.882795    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sw9tn"
Jul 28 19:36:21.883188 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.883175    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 19:36:21.885970 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.885953    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 19:36:21.888159 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.884728    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-173.ec2.internal.18c68b399e16eff8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-173.ec2.internal,UID:ip-10-0-138-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-173.ec2.internal,},FirstTimestamp:2026-07-28 19:36:21.88068044 +0000 UTC m=+0.341439030,LastTimestamp:2026-07-28 19:36:21.88068044 +0000 UTC m=+0.341439030,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-173.ec2.internal,}"
Jul 28 19:36:21.890241 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.890227    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sw9tn"
Jul 28 19:36:21.891036 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891023    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 19:36:21.891100 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891029    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 19:36:21.891185 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.891119    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:21.891625 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891605    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 19:36:21.891707 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891609    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 19:36:21.891707 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891641    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 19:36:21.891811 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891720    2555 factory.go:55] Registering systemd factory
Jul 28 19:36:21.891811 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891736    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 28 19:36:21.891811 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891742    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 19:36:21.891811 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.891750    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 19:36:21.892020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.892008    2555 factory.go:153] Registering CRI-O factory
Jul 28 19:36:21.892020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.892020    2555 factory.go:223] Registration of the crio container factory successfully
Jul 28 19:36:21.892099 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.892065    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 19:36:21.892099 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.892087    2555 factory.go:103] Registering Raw factory
Jul 28 19:36:21.892099 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.892100    2555 manager.go:1196] Started watching for new ooms in manager
Jul 28 19:36:21.892431 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.892422    2555 manager.go:349] Starting recovery of all containers
Jul 28 19:36:21.894290 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.894261    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 19:36:21.894386 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.894338    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 19:36:21.902041 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.902026    2555 manager.go:354] Recovery completed
Jul 28 19:36:21.906141 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.906125    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:36:21.907740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.907725    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:36:21.907819 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.907748    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:36:21.907819 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.907759    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:36:21.908228 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.908215    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 19:36:21.908228 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.908227    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 19:36:21.908322 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.908243    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 19:36:21.908322 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.908229    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:36:21.908903 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.908877    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:36:21.909022 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.909005    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:36:21.909088 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.909024    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:36:21.910248 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.910234    2555 policy_none.go:49] "None policy: Start"
Jul 28 19:36:21.910248 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.910250    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 19:36:21.910352 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.910260    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 19:36:21.912260 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.912248    2555 policy_none.go:47] "Start"
Jul 28 19:36:21.944468 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.944375    2555 manager.go:337] "Starting Device Plugin manager"
Jul 28 19:36:21.944550 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.944487    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 19:36:21.944550 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.944496    2555 server.go:87] "Starting device plugin registration server"
Jul 28 19:36:21.944676 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.944666    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 19:36:21.944734 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.944678    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 19:36:21.944811 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.944785    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 19:36:21.944877 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.944865    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 19:36:21.944994 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.944878    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 19:36:21.945280 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.945261    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 19:36:21.945372 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.945304    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:21.976720 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.976696    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 19:36:21.988382 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.977869    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 19:36:21.988382 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.977922    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 19:36:21.988382 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.977940    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 19:36:21.988382 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:21.977971    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 19:36:21.988382 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:21.985629    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:36:22.045631 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.045613    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:36:22.046305 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.046290    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:36:22.046396 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.046314    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:36:22.046396 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.046324    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:36:22.046396 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.046341    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.058427 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.058371    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.058427 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.058391    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-173.ec2.internal\": node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.073375 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.073356    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.078317 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.078303    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal"]
Jul 28 19:36:22.078372 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.078349    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:36:22.079567 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.079554    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:36:22.079616 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.079579    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:36:22.079616 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.079590    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:36:22.081686 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.081674    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:36:22.081810 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.081795    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.081849 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.081823    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:36:22.082348 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.082329    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:36:22.082348 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.082351    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:36:22.082481 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.082360    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:36:22.082481 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.082394    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:36:22.082481 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.082413    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:36:22.082481 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.082422    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:36:22.084416 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.084401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.084464 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.084433    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:36:22.085135 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.085118    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:36:22.085248 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.085149    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:36:22.085248 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.085162    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:36:22.110634 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.110617    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-173.ec2.internal\" not found" node="ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.115221 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.115205    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-173.ec2.internal\" not found" node="ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.173787 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.173767    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.192760 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.192739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/415ae5ae7d5d3fc074b4ce415469a19e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-173.ec2.internal\" (UID: \"415ae5ae7d5d3fc074b4ce415469a19e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.192869 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.192767    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f5920e338c0b1aa0d84686639273f07-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal\" (UID: \"1f5920e338c0b1aa0d84686639273f07\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.192869 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.192786    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f5920e338c0b1aa0d84686639273f07-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal\" (UID: \"1f5920e338c0b1aa0d84686639273f07\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.274124 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.274099    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.293607 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.293587    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f5920e338c0b1aa0d84686639273f07-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal\" (UID: \"1f5920e338c0b1aa0d84686639273f07\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.293699 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.293615    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/415ae5ae7d5d3fc074b4ce415469a19e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-173.ec2.internal\" (UID: \"415ae5ae7d5d3fc074b4ce415469a19e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.293699 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.293633    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f5920e338c0b1aa0d84686639273f07-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal\" (UID: \"1f5920e338c0b1aa0d84686639273f07\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.293699 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.293673    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f5920e338c0b1aa0d84686639273f07-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal\" (UID: \"1f5920e338c0b1aa0d84686639273f07\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.293833 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.293714    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/415ae5ae7d5d3fc074b4ce415469a19e-config\") pod \"kube-apiserver-proxy-ip-10-0-138-173.ec2.internal\" (UID: \"415ae5ae7d5d3fc074b4ce415469a19e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.293833 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.293718    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f5920e338c0b1aa0d84686639273f07-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal\" (UID: \"1f5920e338c0b1aa0d84686639273f07\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.374678 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.374621    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.414023 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.413984    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.417965 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.417933    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:22.476232 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.475511    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.582316 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.582295    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.685987 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.685927    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.788703 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.788678    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.832829 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.832796    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 19:36:22.833220 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.832971    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 19:36:22.889143 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.889123    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:22.895207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.895170    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 19:31:22 +0000 UTC" deadline="2028-03-16 00:05:51.367503976 +0000 UTC"
Jul 28 19:36:22.895207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.895199    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14308h29m28.47230833s"
Jul 28 19:36:22.898620 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:22.898603    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:36:22.989301 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:22.989244    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:23.038697 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:23.038661    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5920e338c0b1aa0d84686639273f07.slice/crio-b24747d92047e951eaa9f26aa84fb3fcb0d5c06f709dfdc41ac88944f2299fdc WatchSource:0}: Error finding container b24747d92047e951eaa9f26aa84fb3fcb0d5c06f709dfdc41ac88944f2299fdc: Status 404 returned error can't find the container with id b24747d92047e951eaa9f26aa84fb3fcb0d5c06f709dfdc41ac88944f2299fdc
Jul 28 19:36:23.039229 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:23.039209    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415ae5ae7d5d3fc074b4ce415469a19e.slice/crio-8d67d9acb444edccd58a16b4a0d51172a3206ae62040466083918bc4d37bb90e WatchSource:0}: Error finding container 8d67d9acb444edccd58a16b4a0d51172a3206ae62040466083918bc4d37bb90e: Status 404 returned error can't find the container with id 8d67d9acb444edccd58a16b4a0d51172a3206ae62040466083918bc4d37bb90e
Jul 28 19:36:23.041108 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.041092    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:36:23.044999 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.044984    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 19:36:23.091575 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:23.091544    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:23.197308 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:23.197281    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:23.297484 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:23.297434    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:23.398227 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:23.398200    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-173.ec2.internal\" not found"
Jul 28 19:36:23.442718 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.442697    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:36:23.491972 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.491947    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:23.505928 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.505861    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 19:36:23.507456 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.507436    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal"
Jul 28 19:36:23.515895 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.515813    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 19:36:23.822027 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.821994    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:36:23.872572 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.872367    2555 apiserver.go:52] "Watching apiserver"
Jul 28 19:36:23.880549 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.880468    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 19:36:23.881103 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.880980    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-9nkx9","kube-system/konnectivity-agent-2569w","kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal","openshift-dns/node-resolver-67xlq","openshift-multus/network-metrics-daemon-dkh9g","openshift-network-operator/iptables-alerter-f9pp9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9","openshift-cluster-node-tuning-operator/tuned-qcxc5","openshift-image-registry/node-ca-pljjh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal","openshift-multus/multus-4s7gc","openshift-multus/multus-additional-cni-plugins-mj8cx","openshift-network-diagnostics/network-check-target-jfjzl","openshift-ovn-kubernetes/ovnkube-node-dx4p9"]
Jul 28 19:36:23.883979 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.883940    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:23.886195 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.886173    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:23.886291 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:23.886248    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:23.886627 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.886439    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 19:36:23.886627 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.886574    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n8d7f\""
Jul 28 19:36:23.887190 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.887010    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 19:36:23.887190 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.887134    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 19:36:23.889039 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.889018    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.891310 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.891280    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:23.891946 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.891499    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 19:36:23.891946 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.891519    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 19:36:23.891946 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.891776    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 19:36:23.891946 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.891802    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tgck\""
Jul 28 19:36:23.892241 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.891982    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 19:36:23.892241 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.892033    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 19:36:23.892241 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.892134    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 19:36:23.893106 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.893032    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 19:36:23.893833 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.893816    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:23.893930 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:23.893882    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:23.894751 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.894724    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 19:36:23.896079 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.896057    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:23.896774 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.896752    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 19:36:23.896879 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.896796    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6cqm\""
Jul 28 19:36:23.898730 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.898708    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:36:23.898987 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.898970    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 19:36:23.899185 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.899168    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dbm9w\""
Jul 28 19:36:23.899358 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.899340    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 19:36:23.899358 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.899357    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:23.902475 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.902111    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:23.902475 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:23.902173    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:23.902475 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.902286    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2bq4s\""
Jul 28 19:36:23.902475 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.902432    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 19:36:23.903320 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903294    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903455    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903590    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-systemd-units\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903841    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwsx\" (UniqueName: \"kubernetes.io/projected/666e0bbd-3122-4414-b121-88468187b07c-kube-api-access-fkwsx\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903876    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-registration-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903921    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903946    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-log-socket\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903975    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-cni-bin\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.903997    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904017    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xt7s\" (UniqueName: \"kubernetes.io/projected/56cd6a48-ae18-4d17-8258-6a7c488be745-kube-api-access-4xt7s\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904036    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904057    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-kubelet\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904076    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-slash\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904096    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-node-log\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904112    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/210836e2-fdac-4a51-ba2d-7401ef985e49-ovn-node-metrics-cert\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904134    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-var-lib-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.904658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vdg\" (UniqueName: \"kubernetes.io/projected/210836e2-fdac-4a51-ba2d-7401ef985e49-kube-api-access-s6vdg\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904188    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-systemd\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904302    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-ovnkube-config\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-env-overrides\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904431    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666e0bbd-3122-4414-b121-88468187b07c-hosts-file\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904469    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904543    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-etc-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-cni-netd\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904619    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666e0bbd-3122-4414-b121-88468187b07c-tmp-dir\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-sys-fs\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-run-netns\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904839    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904872    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-ovnkube-script-lib\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904967    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6tcp\" (UniqueName: \"kubernetes.io/projected/62542e05-f80b-4490-9213-693a171a19d6-kube-api-access-m6tcp\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.904995    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-socket-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:23.905489 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.905028    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-ovn\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.906290 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.905345    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:23.906290 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.905382    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-device-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:23.907786 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.907766    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:23.909328 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.909305    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 19:36:23.909561 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.909545    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7n6n\""
Jul 28 19:36:23.909744 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.909549    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 19:36:23.910430 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.910410    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:36:23.910667 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.910646    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 19:36:23.911586 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.911199    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bwpl9\""
Jul 28 19:36:23.911586 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.911483    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 19:36:23.911825 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.911797    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:23.914050 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.914029    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:23.914557 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.914516    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 19:36:23.915148 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.915128    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 19:36:23.915237 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.915216    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 19:36:23.915319 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.915303    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 19:36:23.915382 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.915348    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rld2f\""
Jul 28 19:36:23.916432 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.916413    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 19:36:23.917054 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.917034    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9qnp\""
Jul 28 19:36:23.917269 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.917249    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 19:36:23.935602 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.935554    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x2xk8"
Jul 28 19:36:23.943839 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.943813    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x2xk8"
Jul 28 19:36:23.982603 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.982556    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal" event={"ID":"415ae5ae7d5d3fc074b4ce415469a19e","Type":"ContainerStarted","Data":"8d67d9acb444edccd58a16b4a0d51172a3206ae62040466083918bc4d37bb90e"}
Jul 28 19:36:23.983754 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.983730    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal" event={"ID":"1f5920e338c0b1aa0d84686639273f07","Type":"ContainerStarted","Data":"b24747d92047e951eaa9f26aa84fb3fcb0d5c06f709dfdc41ac88944f2299fdc"}
Jul 28 19:36:23.992907 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:23.992882    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 19:36:24.006227 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006205    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:24.006327 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006238    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cnibin\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.006327 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006255    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.006327 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006296    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-cni-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.006327 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006320    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-cni-multus\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.006511 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006347    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/210836e2-fdac-4a51-ba2d-7401ef985e49-ovn-node-metrics-cert\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.006511 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006374    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/373173cb-ee69-427d-a5d6-92018dfddc2f-iptables-alerter-script\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.006511 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006400    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsh72\" (UniqueName: \"kubernetes.io/projected/373173cb-ee69-427d-a5d6-92018dfddc2f-kube-api-access-xsh72\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.006511 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006493    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-system-cni-dir\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.006740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.006740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006569    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-hostroot\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.006740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006597    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-daemon-config\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.006740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006650    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-var-lib-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.006740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006711    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-systemd\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.006740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006732    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-var-lib-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006747    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006781    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-env-overrides\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006809    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-systemd\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666e0bbd-3122-4414-b121-88468187b07c-hosts-file\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/666e0bbd-3122-4414-b121-88468187b07c-hosts-file\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006859    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006947    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.006961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/373173cb-ee69-427d-a5d6-92018dfddc2f-host-slash\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.007026 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007011    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-system-cni-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007064    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-cni-bin\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007123    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-etc-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007149    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-cni-netd\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007174    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666e0bbd-3122-4414-b121-88468187b07c-tmp-dir\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007199    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-sys-fs\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-os-release\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007234    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-cni-netd\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007236    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-etc-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007250    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007295    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-env-overrides\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007304    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6tcp\" (UniqueName: \"kubernetes.io/projected/62542e05-f80b-4490-9213-693a171a19d6-kube-api-access-m6tcp\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-sys-fs\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-socket-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.007455    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:24.007517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007503    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-systemd\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007536    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666e0bbd-3122-4414-b121-88468187b07c-tmp-dir\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.007542    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs podName:62542e05-f80b-4490-9213-693a171a19d6 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:24.507501389 +0000 UTC m=+2.968259983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs") pod "network-metrics-daemon-dkh9g" (UID: "62542e05-f80b-4490-9213-693a171a19d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007562    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-socket-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007613    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a4eb5e7-30db-4811-86f2-6870988e8487-cni-binary-copy\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007643    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-socket-dir-parent\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007690    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-device-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007763    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/674936af-b442-43be-b18d-2677522519c3-serviceca\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007807    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-host\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007841    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-systemd-units\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007882    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-device-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007896    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007952    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysconfig\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007972    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-var-lib-kubelet\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.007979    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-systemd-units\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008005    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9a6e2f7-b600-49dc-8f85-6622afec929e-agent-certs\") pod \"konnectivity-agent-2569w\" (UID: \"f9a6e2f7-b600-49dc-8f85-6622afec929e\") " pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:24.008215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008032    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9a6e2f7-b600-49dc-8f85-6622afec929e-konnectivity-ca\") pod \"konnectivity-agent-2569w\" (UID: \"f9a6e2f7-b600-49dc-8f85-6622afec929e\") " pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008076    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008101    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-etc-selinux\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008113    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8hz8\" (UniqueName: \"kubernetes.io/projected/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-kube-api-access-f8hz8\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008132    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008180    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-lib-modules\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008216    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-cnibin\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008237    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-kubelet\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008280    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-slash\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008310    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-node-log\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-kubelet\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008375    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-conf-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008375    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-node-log\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008424    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8jmm\" (UniqueName: \"kubernetes.io/projected/6a4eb5e7-30db-4811-86f2-6870988e8487-kube-api-access-n8jmm\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008457    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vdg\" (UniqueName: \"kubernetes.io/projected/210836e2-fdac-4a51-ba2d-7401ef985e49-kube-api-access-s6vdg\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008483    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/674936af-b442-43be-b18d-2677522519c3-host\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.009058 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008523    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-sys\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008562    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxdn\" (UniqueName: \"kubernetes.io/projected/c09bf96e-35ff-4dac-9a91-1267e467da6a-kube-api-access-vtxdn\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008589    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-k8s-cni-cncf-io\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008615    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-netns\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008639    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-kubelet\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008676    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-multus-certs\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008736    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-ovnkube-config\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mbd9\" (UniqueName: \"kubernetes.io/projected/674936af-b442-43be-b18d-2677522519c3-kube-api-access-6mbd9\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008788    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-etc-kubernetes\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008814    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-os-release\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008847    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/464229a3-6aff-4f24-ba21-64b65a2182a4-dbus\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008873    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008899    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-run-netns\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-slash\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.008925    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009030    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-ovnkube-script-lib\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009061    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-openvswitch\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.009893 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009072    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-modprobe-d\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009079    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-run-netns\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009118    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysctl-conf\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009254    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c09bf96e-35ff-4dac-9a91-1267e467da6a-tmp\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009291    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-ovn\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009362    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009395    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/464229a3-6aff-4f24-ba21-64b65a2182a4-kubelet-config\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009421    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-run-ovn\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009435    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-kubernetes\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009463    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysctl-d\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009543    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-run-ovn-kubernetes\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-ovnkube-config\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009604    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/210836e2-fdac-4a51-ba2d-7401ef985e49-ovnkube-script-lib\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009585    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-tuned\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009711    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwsx\" (UniqueName: \"kubernetes.io/projected/666e0bbd-3122-4414-b121-88468187b07c-kube-api-access-fkwsx\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009764    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-registration-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009856    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.010700 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009929    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-run\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.011477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.009970    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-log-socket\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.011477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.010031    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-cni-bin\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.011477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.010067    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xt7s\" (UniqueName: \"kubernetes.io/projected/56cd6a48-ae18-4d17-8258-6a7c488be745-kube-api-access-4xt7s\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.011477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.010149    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56cd6a48-ae18-4d17-8258-6a7c488be745-registration-dir\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.011477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.010177    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-log-socket\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.011477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.010338    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/210836e2-fdac-4a51-ba2d-7401ef985e49-host-cni-bin\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.011477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.010962    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/210836e2-fdac-4a51-ba2d-7401ef985e49-ovn-node-metrics-cert\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.014309 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.014290    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:36:24.014400 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.014317    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:36:24.014400 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.014331    2555 projected.go:196] Error preparing data for projected volume kube-api-access-v2mnz for pod openshift-network-diagnostics/network-check-target-jfjzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:24.014400 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.014389    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz podName:6af25eb8-7d06-493e-8d26-343b128261e1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:24.514370555 +0000 UTC m=+2.975129149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v2mnz" (UniqueName: "kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz") pod "network-check-target-jfjzl" (UID: "6af25eb8-7d06-493e-8d26-343b128261e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:24.021765 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.021735    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6tcp\" (UniqueName: \"kubernetes.io/projected/62542e05-f80b-4490-9213-693a171a19d6-kube-api-access-m6tcp\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:24.023535 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.023514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vdg\" (UniqueName: \"kubernetes.io/projected/210836e2-fdac-4a51-ba2d-7401ef985e49-kube-api-access-s6vdg\") pod \"ovnkube-node-dx4p9\" (UID: \"210836e2-fdac-4a51-ba2d-7401ef985e49\") " pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.023721 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.023697    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwsx\" (UniqueName: \"kubernetes.io/projected/666e0bbd-3122-4414-b121-88468187b07c-kube-api-access-fkwsx\") pod \"node-resolver-67xlq\" (UID: \"666e0bbd-3122-4414-b121-88468187b07c\") " pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:24.023943 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.023885    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xt7s\" (UniqueName: \"kubernetes.io/projected/56cd6a48-ae18-4d17-8258-6a7c488be745-kube-api-access-4xt7s\") pod \"aws-ebs-csi-driver-node-lqfn9\" (UID: \"56cd6a48-ae18-4d17-8258-6a7c488be745\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.110417 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110350    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-systemd\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.110417 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110391    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a4eb5e7-30db-4811-86f2-6870988e8487-cni-binary-copy\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.110579 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110417    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-socket-dir-parent\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.110579 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/674936af-b442-43be-b18d-2677522519c3-serviceca\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.110579 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110484    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-systemd\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.110579 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110487    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-host\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.110579 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110542    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-host\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.110579 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110563    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-socket-dir-parent\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysconfig\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-var-lib-kubelet\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110662    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9a6e2f7-b600-49dc-8f85-6622afec929e-agent-certs\") pod \"konnectivity-agent-2569w\" (UID: \"f9a6e2f7-b600-49dc-8f85-6622afec929e\") " pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110687    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9a6e2f7-b600-49dc-8f85-6622afec929e-konnectivity-ca\") pod \"konnectivity-agent-2569w\" (UID: \"f9a6e2f7-b600-49dc-8f85-6622afec929e\") " pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110695    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysconfig\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8hz8\" (UniqueName: \"kubernetes.io/projected/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-kube-api-access-f8hz8\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-lib-modules\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110763    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-cnibin\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110806    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110841    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-conf-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110865    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8jmm\" (UniqueName: \"kubernetes.io/projected/6a4eb5e7-30db-4811-86f2-6870988e8487-kube-api-access-n8jmm\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.110896 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110897    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/674936af-b442-43be-b18d-2677522519c3-host\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/674936af-b442-43be-b18d-2677522519c3-serviceca\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110937    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-sys\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110962    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxdn\" (UniqueName: \"kubernetes.io/projected/c09bf96e-35ff-4dac-9a91-1267e467da6a-kube-api-access-vtxdn\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110984    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a4eb5e7-30db-4811-86f2-6870988e8487-cni-binary-copy\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111029    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-k8s-cni-cncf-io\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111075    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-conf-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.110988    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-k8s-cni-cncf-io\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111131    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-netns\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111163    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/674936af-b442-43be-b18d-2677522519c3-host\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111181    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-sys\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111192    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-kubelet\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111228    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-kubelet\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111244    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-multus-certs\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111283    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-multus-certs\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111211    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-run-netns\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111156    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-var-lib-kubelet\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111299    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mbd9\" (UniqueName: \"kubernetes.io/projected/674936af-b442-43be-b18d-2677522519c3-kube-api-access-6mbd9\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.111539 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111359    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-cnibin\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-etc-kubernetes\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-os-release\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111458    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/464229a3-6aff-4f24-ba21-64b65a2182a4-dbus\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111465    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-etc-kubernetes\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111425    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-lib-modules\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-os-release\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-modprobe-d\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.111603    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111608    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/464229a3-6aff-4f24-ba21-64b65a2182a4-dbus\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111626    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysctl-conf\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.111658    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret podName:464229a3-6aff-4f24-ba21-64b65a2182a4 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:24.611640208 +0000 UTC m=+3.072398801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret") pod "global-pull-secret-syncer-9nkx9" (UID: "464229a3-6aff-4f24-ba21-64b65a2182a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111679    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c09bf96e-35ff-4dac-9a91-1267e467da6a-tmp\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111702    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-modprobe-d\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111713    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/464229a3-6aff-4f24-ba21-64b65a2182a4-kubelet-config\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.112460 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111741    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-kubernetes\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111767    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysctl-d\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111783    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysctl-conf\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111791    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-tuned\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9a6e2f7-b600-49dc-8f85-6622afec929e-konnectivity-ca\") pod \"konnectivity-agent-2569w\" (UID: \"f9a6e2f7-b600-49dc-8f85-6622afec929e\") " pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-run\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111862    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/464229a3-6aff-4f24-ba21-64b65a2182a4-kubelet-config\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-kubernetes\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cnibin\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-run\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.111990    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-sysctl-d\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112019    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112032    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cnibin\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112081    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-cni-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112127    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-cni-multus\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112157    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/373173cb-ee69-427d-a5d6-92018dfddc2f-iptables-alerter-script\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.113356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsh72\" (UniqueName: \"kubernetes.io/projected/373173cb-ee69-427d-a5d6-92018dfddc2f-kube-api-access-xsh72\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112208    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-system-cni-dir\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112234    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112246    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-cni-multus\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112258    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-hostroot\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112316    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-cni-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112355    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-hostroot\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-daemon-config\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112402    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112489    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-cni-binary-copy\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112491    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-system-cni-dir\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112563    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/373173cb-ee69-427d-a5d6-92018dfddc2f-host-slash\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-system-cni-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112616    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-cni-bin\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112628    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-system-cni-dir\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112644    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-os-release\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112676    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-host-var-lib-cni-bin\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.114111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112685    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/373173cb-ee69-427d-a5d6-92018dfddc2f-host-slash\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.115127 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112741    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a4eb5e7-30db-4811-86f2-6870988e8487-os-release\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.115127 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112912    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6a4eb5e7-30db-4811-86f2-6870988e8487-multus-daemon-config\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.115127 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.112968    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/373173cb-ee69-427d-a5d6-92018dfddc2f-iptables-alerter-script\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.115127 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.114294    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c09bf96e-35ff-4dac-9a91-1267e467da6a-tmp\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.115127 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.114888    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c09bf96e-35ff-4dac-9a91-1267e467da6a-etc-tuned\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.115127 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.115117    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9a6e2f7-b600-49dc-8f85-6622afec929e-agent-certs\") pod \"konnectivity-agent-2569w\" (UID: \"f9a6e2f7-b600-49dc-8f85-6622afec929e\") " pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:24.129999 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.129975    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mbd9\" (UniqueName: \"kubernetes.io/projected/674936af-b442-43be-b18d-2677522519c3-kube-api-access-6mbd9\") pod \"node-ca-pljjh\" (UID: \"674936af-b442-43be-b18d-2677522519c3\") " pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.130403 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.130379    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8jmm\" (UniqueName: \"kubernetes.io/projected/6a4eb5e7-30db-4811-86f2-6870988e8487-kube-api-access-n8jmm\") pod \"multus-4s7gc\" (UID: \"6a4eb5e7-30db-4811-86f2-6870988e8487\") " pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.130630 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.130606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsh72\" (UniqueName: \"kubernetes.io/projected/373173cb-ee69-427d-a5d6-92018dfddc2f-kube-api-access-xsh72\") pod \"iptables-alerter-f9pp9\" (UID: \"373173cb-ee69-427d-a5d6-92018dfddc2f\") " pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.130954 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.130936    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxdn\" (UniqueName: \"kubernetes.io/projected/c09bf96e-35ff-4dac-9a91-1267e467da6a-kube-api-access-vtxdn\") pod \"tuned-qcxc5\" (UID: \"c09bf96e-35ff-4dac-9a91-1267e467da6a\") " pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.130954 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.130941    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8hz8\" (UniqueName: \"kubernetes.io/projected/eddbfddc-1f2d-4857-8ad5-698e5fcf4d77-kube-api-access-f8hz8\") pod \"multus-additional-cni-plugins-mj8cx\" (UID: \"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77\") " pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.197225 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.197204    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9"
Jul 28 19:36:24.205949 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.205929    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:24.218055 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.218032    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-67xlq"
Jul 28 19:36:24.224873 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.224857    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f9pp9"
Jul 28 19:36:24.232760 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.232742    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pljjh"
Jul 28 19:36:24.240560 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.240538    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:24.248480 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.248461    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qcxc5"
Jul 28 19:36:24.255382 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.255359    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4s7gc"
Jul 28 19:36:24.263199 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.263177    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mj8cx"
Jul 28 19:36:24.514689 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.514615    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:24.514689 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.514684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:24.514889 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.514781    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:24.514889 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.514792    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:36:24.514889 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.514812    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:36:24.514889 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.514824    2555 projected.go:196] Error preparing data for projected volume kube-api-access-v2mnz for pod openshift-network-diagnostics/network-check-target-jfjzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:24.514889 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.514854    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs podName:62542e05-f80b-4490-9213-693a171a19d6 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:25.514834745 +0000 UTC m=+3.975593337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs") pod "network-metrics-daemon-dkh9g" (UID: "62542e05-f80b-4490-9213-693a171a19d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:24.514889 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.514871    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz podName:6af25eb8-7d06-493e-8d26-343b128261e1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:25.514863837 +0000 UTC m=+3.975622417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2mnz" (UniqueName: "kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz") pod "network-check-target-jfjzl" (UID: "6af25eb8-7d06-493e-8d26-343b128261e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:24.615369 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.615340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:24.615539 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.615504    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:24.615596 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.615570    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret podName:464229a3-6aff-4f24-ba21-64b65a2182a4 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:25.615553577 +0000 UTC m=+4.076312156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret") pod "global-pull-secret-syncer-9nkx9" (UID: "464229a3-6aff-4f24-ba21-64b65a2182a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:24.946885 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.946804    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:31:24 +0000 UTC" deadline="2028-04-21 06:49:34.606058646 +0000 UTC"
Jul 28 19:36:24.946885 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.946841    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15179h13m9.659222162s"
Jul 28 19:36:24.978702 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:24.978675    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:24.978840 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:24.978783    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:25.003569 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:25.003490    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210836e2_fdac_4a51_ba2d_7401ef985e49.slice/crio-33f21885feb9fbed81ba466fa0018af68588d397fa6a147e648d096d14cfc95c WatchSource:0}: Error finding container 33f21885feb9fbed81ba466fa0018af68588d397fa6a147e648d096d14cfc95c: Status 404 returned error can't find the container with id 33f21885feb9fbed81ba466fa0018af68588d397fa6a147e648d096d14cfc95c
Jul 28 19:36:25.008741 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:25.008718    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674936af_b442_43be_b18d_2677522519c3.slice/crio-58de15faa8688ce2e3dc2e80252f556c6db577aa530280fdb6c153e6f9965882 WatchSource:0}: Error finding container 58de15faa8688ce2e3dc2e80252f556c6db577aa530280fdb6c153e6f9965882: Status 404 returned error can't find the container with id 58de15faa8688ce2e3dc2e80252f556c6db577aa530280fdb6c153e6f9965882
Jul 28 19:36:25.009508 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:25.009472    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cd6a48_ae18_4d17_8258_6a7c488be745.slice/crio-26e3c3a4ad2422f137b6b7745b082a6ffec545c07cb9623a4a7a5b9b02d6b25f WatchSource:0}: Error finding container 26e3c3a4ad2422f137b6b7745b082a6ffec545c07cb9623a4a7a5b9b02d6b25f: Status 404 returned error can't find the container with id 26e3c3a4ad2422f137b6b7745b082a6ffec545c07cb9623a4a7a5b9b02d6b25f
Jul 28 19:36:25.010366 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:25.010294    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666e0bbd_3122_4414_b121_88468187b07c.slice/crio-74d6823c2ea6fb1df92af2ad298b39845fd70f35741dca0afaa7f53cda8c84a0 WatchSource:0}: Error finding container 74d6823c2ea6fb1df92af2ad298b39845fd70f35741dca0afaa7f53cda8c84a0: Status 404 returned error can't find the container with id 74d6823c2ea6fb1df92af2ad298b39845fd70f35741dca0afaa7f53cda8c84a0
Jul 28 19:36:25.011046 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:25.010955    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a6e2f7_b600_49dc_8f85_6622afec929e.slice/crio-3c8595de65a8fb0051952afd7b20b163649b08eda67aa8b97caf3ff7769e21a7 WatchSource:0}: Error finding container 3c8595de65a8fb0051952afd7b20b163649b08eda67aa8b97caf3ff7769e21a7: Status 404 returned error can't find the container with id 3c8595de65a8fb0051952afd7b20b163649b08eda67aa8b97caf3ff7769e21a7
Jul 28 19:36:25.524420 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:25.524182    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:25.524420 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:25.524398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:25.524643 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.524523    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:25.524643 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.524583    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs podName:62542e05-f80b-4490-9213-693a171a19d6 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:27.524563249 +0000 UTC m=+5.985321841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs") pod "network-metrics-daemon-dkh9g" (UID: "62542e05-f80b-4490-9213-693a171a19d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:25.525164 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.525019    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:36:25.525164 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.525043    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:36:25.525164 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.525058    2555 projected.go:196] Error preparing data for projected volume kube-api-access-v2mnz for pod openshift-network-diagnostics/network-check-target-jfjzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:25.525164 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.525108    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz podName:6af25eb8-7d06-493e-8d26-343b128261e1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:27.525090046 +0000 UTC m=+5.985848643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2mnz" (UniqueName: "kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz") pod "network-check-target-jfjzl" (UID: "6af25eb8-7d06-493e-8d26-343b128261e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:25.624805 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:25.624767    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:25.624979 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.624966    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:25.625048 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.625029    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret podName:464229a3-6aff-4f24-ba21-64b65a2182a4 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:27.62500976 +0000 UTC m=+6.085768345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret") pod "global-pull-secret-syncer-9nkx9" (UID: "464229a3-6aff-4f24-ba21-64b65a2182a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:25.948098 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:25.947985    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:31:24 +0000 UTC" deadline="2028-02-20 16:58:49.806909036 +0000 UTC"
Jul 28 19:36:25.948098 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:25.948052    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13725h22m23.858861931s"
Jul 28 19:36:25.981657 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:25.981628    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:25.981814 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.981756    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:25.982219 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:25.982193    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:25.982332 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:25.982310    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:26.010919 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.007678    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2569w" event={"ID":"f9a6e2f7-b600-49dc-8f85-6622afec929e","Type":"ContainerStarted","Data":"3c8595de65a8fb0051952afd7b20b163649b08eda67aa8b97caf3ff7769e21a7"}
Jul 28 19:36:26.017545 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.017489    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-67xlq" event={"ID":"666e0bbd-3122-4414-b121-88468187b07c","Type":"ContainerStarted","Data":"74d6823c2ea6fb1df92af2ad298b39845fd70f35741dca0afaa7f53cda8c84a0"}
Jul 28 19:36:26.027956 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.027871    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9" event={"ID":"56cd6a48-ae18-4d17-8258-6a7c488be745","Type":"ContainerStarted","Data":"26e3c3a4ad2422f137b6b7745b082a6ffec545c07cb9623a4a7a5b9b02d6b25f"}
Jul 28 19:36:26.039282 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.039242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pljjh" event={"ID":"674936af-b442-43be-b18d-2677522519c3","Type":"ContainerStarted","Data":"58de15faa8688ce2e3dc2e80252f556c6db577aa530280fdb6c153e6f9965882"}
Jul 28 19:36:26.052039 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.052011    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal" event={"ID":"1f5920e338c0b1aa0d84686639273f07","Type":"ContainerDied","Data":"95e7f47fb0f224bfc9ea66e0763b3eb2a20f5926ff13b9629cb7d57d5e4eaa57"}
Jul 28 19:36:26.052566 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.051882    2555 generic.go:356] "Generic (PLEG): container finished" podID="1f5920e338c0b1aa0d84686639273f07" containerID="95e7f47fb0f224bfc9ea66e0763b3eb2a20f5926ff13b9629cb7d57d5e4eaa57" exitCode=0
Jul 28 19:36:26.059936 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.059844    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f9pp9" event={"ID":"373173cb-ee69-427d-a5d6-92018dfddc2f","Type":"ContainerStarted","Data":"14cd2e7a4fc9f203d15071ec8c143abc0d9acc4636d1b51f2c8a9197872563ae"}
Jul 28 19:36:26.067383 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.067360    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"33f21885feb9fbed81ba466fa0018af68588d397fa6a147e648d096d14cfc95c"}
Jul 28 19:36:26.089916 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.089875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal" event={"ID":"415ae5ae7d5d3fc074b4ce415469a19e","Type":"ContainerStarted","Data":"a3016718c468864f0957a9ea632df160fa15f705f92cd91ab793ab00705e3708"}
Jul 28 19:36:26.113177 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.113151    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4s7gc" event={"ID":"6a4eb5e7-30db-4811-86f2-6870988e8487","Type":"ContainerStarted","Data":"4173db40161a71023b0a74914ea141d6a099a4ea6144915c9584a69d1dca734f"}
Jul 28 19:36:26.121707 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.121683    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qcxc5" event={"ID":"c09bf96e-35ff-4dac-9a91-1267e467da6a","Type":"ContainerStarted","Data":"451d5096e33b93415402851a3b0c52ef5577cffd218bb9facf12fc195217b5b7"}
Jul 28 19:36:26.126459 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.126437    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" event={"ID":"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77","Type":"ContainerStarted","Data":"83362508b0d830c955336c0056c672b905180074dd030adf48762883f14dfd4e"}
Jul 28 19:36:26.979392 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:26.978867    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:26.979392 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:26.979024    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:27.144984 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:27.144942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal" event={"ID":"1f5920e338c0b1aa0d84686639273f07","Type":"ContainerStarted","Data":"d3568c085ba0c8d425bf5244786eafcdb12fd76ea1715a4e9d2e4857744399bc"}
Jul 28 19:36:27.160550 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:27.160500    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-173.ec2.internal" podStartSLOduration=4.160482227 podStartE2EDuration="4.160482227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:26.104533327 +0000 UTC m=+4.565291929" watchObservedRunningTime="2026-07-28 19:36:27.160482227 +0000 UTC m=+5.621240827"
Jul 28 19:36:27.538863 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:27.538069    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:27.538863 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:27.538135    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:27.538863 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.538279    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:27.538863 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.538340    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs podName:62542e05-f80b-4490-9213-693a171a19d6 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:31.538321859 +0000 UTC m=+9.999080446 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs") pod "network-metrics-daemon-dkh9g" (UID: "62542e05-f80b-4490-9213-693a171a19d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:27.538863 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.538753    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:36:27.538863 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.538772    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:36:27.538863 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.538785    2555 projected.go:196] Error preparing data for projected volume kube-api-access-v2mnz for pod openshift-network-diagnostics/network-check-target-jfjzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:27.538863 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.538831    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz podName:6af25eb8-7d06-493e-8d26-343b128261e1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:31.538814506 +0000 UTC m=+9.999573085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2mnz" (UniqueName: "kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz") pod "network-check-target-jfjzl" (UID: "6af25eb8-7d06-493e-8d26-343b128261e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:27.639948 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:27.639387    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:27.639948 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.639510    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:27.639948 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.639567    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret podName:464229a3-6aff-4f24-ba21-64b65a2182a4 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:31.639550216 +0000 UTC m=+10.100308805 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret") pod "global-pull-secret-syncer-9nkx9" (UID: "464229a3-6aff-4f24-ba21-64b65a2182a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:27.980314 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:27.978156    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:27.980314 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.978278    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:27.980314 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:27.978632    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:27.980314 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:27.978774    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:28.978725 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:28.978279    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:28.978725 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:28.978423    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:29.980183 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:29.979048    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:29.980183 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:29.979073    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:29.980183 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:29.979257    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:29.980183 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:29.979223    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:30.978681 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:30.978646    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:30.979104 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:30.978851    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:31.571847 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:31.571816    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:31.572318 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:31.571875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:31.572318 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.572008    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:31.572318 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.572024    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:36:31.572318 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.572051    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:36:31.572318 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.572064    2555 projected.go:196] Error preparing data for projected volume kube-api-access-v2mnz for pod openshift-network-diagnostics/network-check-target-jfjzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:31.572318 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.572072    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs podName:62542e05-f80b-4490-9213-693a171a19d6 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:39.572053271 +0000 UTC m=+18.032811851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs") pod "network-metrics-daemon-dkh9g" (UID: "62542e05-f80b-4490-9213-693a171a19d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:31.572318 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.572114    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz podName:6af25eb8-7d06-493e-8d26-343b128261e1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:39.572098243 +0000 UTC m=+18.032856839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2mnz" (UniqueName: "kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz") pod "network-check-target-jfjzl" (UID: "6af25eb8-7d06-493e-8d26-343b128261e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:31.673034 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:31.672996    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:31.673202 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.673168    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:31.673265 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.673222    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret podName:464229a3-6aff-4f24-ba21-64b65a2182a4 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:39.673204753 +0000 UTC m=+18.133963332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret") pod "global-pull-secret-syncer-9nkx9" (UID: "464229a3-6aff-4f24-ba21-64b65a2182a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:31.979394 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:31.979320    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:31.979545 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.979404    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:31.979545 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:31.979492    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:31.979683 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:31.979573    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:32.978444 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:32.978408    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:32.978829 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:32.978556    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:33.978477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:33.978320    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:33.978477 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:33.978441    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:33.979062 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:33.978956    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:33.979107 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:33.979058    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:34.978501 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:34.978468    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:34.978901 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:34.978576    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:35.978924 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:35.978787    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:35.979347 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:35.978928    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:35.979347 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:35.978978    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:35.979347 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:35.979071    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:36.978510 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:36.978484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:36.978667 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:36.978587    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:37.978939 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:37.978904    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:37.978939 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:37.978930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:37.979468 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:37.979022    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:37.979468 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:37.979136    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:38.978347 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:38.978312    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:38.978557 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:38.978450    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:39.633742 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:39.633710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:39.634109 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:39.633751    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:39.634109 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.633841    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:39.634109 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.633861    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:36:39.634109 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.633882    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:36:39.634109 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.633903    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs podName:62542e05-f80b-4490-9213-693a171a19d6 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:55.633875047 +0000 UTC m=+34.094633624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs") pod "network-metrics-daemon-dkh9g" (UID: "62542e05-f80b-4490-9213-693a171a19d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:39.634109 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.633908    2555 projected.go:196] Error preparing data for projected volume kube-api-access-v2mnz for pod openshift-network-diagnostics/network-check-target-jfjzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:39.634109 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.633953    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz podName:6af25eb8-7d06-493e-8d26-343b128261e1 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:55.633939968 +0000 UTC m=+34.094698545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2mnz" (UniqueName: "kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz") pod "network-check-target-jfjzl" (UID: "6af25eb8-7d06-493e-8d26-343b128261e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:39.734763 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:39.734736    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:39.734939 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.734853    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:39.734939 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.734931    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret podName:464229a3-6aff-4f24-ba21-64b65a2182a4 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:55.734912133 +0000 UTC m=+34.195670713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret") pod "global-pull-secret-syncer-9nkx9" (UID: "464229a3-6aff-4f24-ba21-64b65a2182a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:39.978348 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:39.978258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:39.978508 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:39.978259    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:39.978508 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.978379    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:39.978508 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:39.978480    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:40.979063 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:40.979032    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:40.979483 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:40.979145    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:41.979724 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:41.979693    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:41.980183 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:41.979793    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:41.980183 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:41.979880    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:41.980183 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:41.980013    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:42.978182 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:42.978150    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:42.978350 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:42.978273    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:43.979281 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:43.978867    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:43.980100 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:43.978870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:43.980100 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:43.979375    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:43.980100 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:43.979520    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:44.178136 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.178064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-67xlq" event={"ID":"666e0bbd-3122-4414-b121-88468187b07c","Type":"ContainerStarted","Data":"0c83428b32b3df7d5e221dc7ff1cca5ae4e767b6f0adc31e768ed2f68b253230"}
Jul 28 19:36:44.180498 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.180465    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9" event={"ID":"56cd6a48-ae18-4d17-8258-6a7c488be745","Type":"ContainerStarted","Data":"46fc42e287a6ff67e91c0c9730a068ccd4e86813f9632033dc865ffc7749fddb"}
Jul 28 19:36:44.183067 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.183039    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pljjh" event={"ID":"674936af-b442-43be-b18d-2677522519c3","Type":"ContainerStarted","Data":"7452b3dd37e5b693954e04deef8e0ccafc8e488b4f2db32edf939096ebf3c34c"}
Jul 28 19:36:44.186316 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.186290    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"bd5cdb834f5fc4d13580243b64628999fc27e7558863917de2a5d12af0220ba9"}
Jul 28 19:36:44.186407 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.186350    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"e5ae5fdcf787643389f43f88edd5b8c6cf86b487d4f8d3d2b20dc587de670951"}
Jul 28 19:36:44.186407 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.186393    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"a106f30f7d02bdc947ad64986115629d7e6288a76beab5a49fd14d53fa87581d"}
Jul 28 19:36:44.186517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.186435    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"117d1b9724e260284e2f3b94058ec925e06f39d98eefb9ea2a1440427551234f"}
Jul 28 19:36:44.186517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.186478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"f11b89a1738da4cd9dc6c06441b0fd23a68fb88eabd69100cbfb67ca60f5b76c"}
Jul 28 19:36:44.186517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.186512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"8d8a03ddeebd23a86c170ae743d008dc7ffc2f3e09f58b4cc1bab89494d7fa22"}
Jul 28 19:36:44.187780 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.187756    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4s7gc" event={"ID":"6a4eb5e7-30db-4811-86f2-6870988e8487","Type":"ContainerStarted","Data":"8ba58463a82e0d70d358ad714c0a255292371ef016328d690ca1e76c11705366"}
Jul 28 19:36:44.189378 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.189350    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qcxc5" event={"ID":"c09bf96e-35ff-4dac-9a91-1267e467da6a","Type":"ContainerStarted","Data":"3b9de18fad303c0ecc2e775f7a1d29cb49c6958dd48bbf6c7e62d7ca9c5fe79a"}
Jul 28 19:36:44.190935 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.190916    2555 generic.go:356] "Generic (PLEG): container finished" podID="eddbfddc-1f2d-4857-8ad5-698e5fcf4d77" containerID="1ec1964a90a858da87245a129426aaa0359d345c4bee31baf13b50cd00373d3f" exitCode=0
Jul 28 19:36:44.191031 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.190982    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" event={"ID":"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77","Type":"ContainerDied","Data":"1ec1964a90a858da87245a129426aaa0359d345c4bee31baf13b50cd00373d3f"}
Jul 28 19:36:44.192557 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.192512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2569w" event={"ID":"f9a6e2f7-b600-49dc-8f85-6622afec929e","Type":"ContainerStarted","Data":"73162e469902ac8d42ff87db50e64f1130c1c636c142ae7136c20e9cc9e5d1d5"}
Jul 28 19:36:44.193354 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.193316    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-173.ec2.internal" podStartSLOduration=21.19330598 podStartE2EDuration="21.19330598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:27.161162539 +0000 UTC m=+5.621921140" watchObservedRunningTime="2026-07-28 19:36:44.19330598 +0000 UTC m=+22.654064570"
Jul 28 19:36:44.193839 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.193783    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-67xlq" podStartSLOduration=3.942119947 podStartE2EDuration="22.193772734s" totalImagesPullingTime="18.251652787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:44.193262642 +0000 UTC m=+22.654021251" watchObservedRunningTime="2026-07-28 19:36:44.193772734 +0000 UTC m=+22.654531332"
Jul 28 19:36:44.207785 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.207745    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2569w" podStartSLOduration=7.580812669 podStartE2EDuration="22.207729734s" totalImagesPullingTime="14.626917065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:44.207078046 +0000 UTC m=+22.667836644" watchObservedRunningTime="2026-07-28 19:36:44.207729734 +0000 UTC m=+22.668488334"
Jul 28 19:36:44.223171 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.223140    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qcxc5" podStartSLOduration=3.959394434 podStartE2EDuration="22.223129539s" totalImagesPullingTime="18.263735105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:44.223048811 +0000 UTC m=+22.683807407" watchObservedRunningTime="2026-07-28 19:36:44.223129539 +0000 UTC m=+22.683888138"
Jul 28 19:36:44.239724 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.239686    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4s7gc" podStartSLOduration=3.941184261 podStartE2EDuration="22.23967471s" totalImagesPullingTime="18.298490449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:44.239632015 +0000 UTC m=+22.700390614" watchObservedRunningTime="2026-07-28 19:36:44.23967471 +0000 UTC m=+22.700433309"
Jul 28 19:36:44.260252 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.260219    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pljjh" podStartSLOduration=4.00576076 podStartE2EDuration="22.260210318s" totalImagesPullingTime="18.254449558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:44.259878011 +0000 UTC m=+22.720636609" watchObservedRunningTime="2026-07-28 19:36:44.260210318 +0000 UTC m=+22.720968917"
Jul 28 19:36:44.702306 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.702285    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 19:36:44.959159 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.959044    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T19:36:44.702301322Z","UUID":"22a944b6-a7c8-4b2c-82e5-aa77120d6199","Handler":null,"Name":"","Endpoint":""}
Jul 28 19:36:44.960717 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.960694    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 19:36:44.960823 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.960731    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 19:36:44.978724 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:44.978707    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:44.978847 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:44.978825    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:45.196425 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:45.196387    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9" event={"ID":"56cd6a48-ae18-4d17-8258-6a7c488be745","Type":"ContainerStarted","Data":"e93e8d34fa1aff5957ed31972deff2d88c30c0a660ca6c62ea25caecb39ce1ac"}
Jul 28 19:36:45.197978 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:45.197926    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f9pp9" event={"ID":"373173cb-ee69-427d-a5d6-92018dfddc2f","Type":"ContainerStarted","Data":"da4543fbb363baaf34856325c14af6eaebbec188107ae8959a203380fba50e11"}
Jul 28 19:36:45.213556 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:45.213476    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f9pp9" podStartSLOduration=4.96789139 podStartE2EDuration="23.213459349s" totalImagesPullingTime="18.245567959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:45.212810181 +0000 UTC m=+23.673568793" watchObservedRunningTime="2026-07-28 19:36:45.213459349 +0000 UTC m=+23.674217950"
Jul 28 19:36:45.978901 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:45.978861    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:45.979014 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:45.978865    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:45.979074 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:45.979006    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:45.979134 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:45.979074    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:46.205334 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:46.205256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"ba2abe0f92bfad4f61ad07e64ef0008106f0f1e8101e599c47a8679340b19d58"}
Jul 28 19:36:46.207740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:46.207703    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9" event={"ID":"56cd6a48-ae18-4d17-8258-6a7c488be745","Type":"ContainerStarted","Data":"3a052c0dc3077264c3d180119feb1bbc8e5cc52c9eaf8fd7f80cb8bd098099c8"}
Jul 28 19:36:46.226373 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:46.226328    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lqfn9" podStartSLOduration=3.7435178369999997 podStartE2EDuration="24.226310729s" totalImagesPullingTime="20.482792892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:46.226282881 +0000 UTC m=+24.687041486" watchObservedRunningTime="2026-07-28 19:36:46.226310729 +0000 UTC m=+24.687069321"
Jul 28 19:36:46.978429 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:46.978274    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:46.978659 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:46.978525    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:47.208275 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:47.208247    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:47.208959 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:47.208935    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:47.979142 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:47.979102    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:47.979330 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:47.979278    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:47.979403 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:47.979328    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:47.979458 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:47.979422    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:48.979176 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:48.978967    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:48.979771 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:48.979200    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:49.217328 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:49.217296    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" event={"ID":"210836e2-fdac-4a51-ba2d-7401ef985e49","Type":"ContainerStarted","Data":"b66a74e87b14dc759b81fc2e0474cdb6ca4906cc8b5ccbc268402e67d91b7b0e"}
Jul 28 19:36:49.217514 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:49.217489    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:49.219196 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:49.219170    2555 generic.go:356] "Generic (PLEG): container finished" podID="eddbfddc-1f2d-4857-8ad5-698e5fcf4d77" containerID="b85e9e1d84e24bcf7f13b9bcd4b8d33be4d3588772bc467ac46ce41b8bc37e45" exitCode=0
Jul 28 19:36:49.219310 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:49.219211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" event={"ID":"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77","Type":"ContainerDied","Data":"b85e9e1d84e24bcf7f13b9bcd4b8d33be4d3588772bc467ac46ce41b8bc37e45"}
Jul 28 19:36:49.232732 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:49.232691    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:49.244974 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:49.244938    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9" podStartSLOduration=8.899294202 podStartE2EDuration="27.244926425s" totalImagesPullingTime="18.345632223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:49.244630087 +0000 UTC m=+27.705388707" watchObservedRunningTime="2026-07-28 19:36:49.244926425 +0000 UTC m=+27.705685023"
Jul 28 19:36:49.979170 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:49.979142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:49.979411 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:49.979273    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:49.979411 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:49.979313    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:49.979411 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:49.979398    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:50.223674 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.223608    2555 generic.go:356] "Generic (PLEG): container finished" podID="eddbfddc-1f2d-4857-8ad5-698e5fcf4d77" containerID="4b7574210550cd218c1d7c7bd36b492e089272195857a809217c938740a8e519" exitCode=0
Jul 28 19:36:50.223815 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.223665    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" event={"ID":"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77","Type":"ContainerDied","Data":"4b7574210550cd218c1d7c7bd36b492e089272195857a809217c938740a8e519"}
Jul 28 19:36:50.224174 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.224147    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:50.224299 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.224225    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:50.241201 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.241181    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:36:50.431166 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.431135    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9nkx9"]
Jul 28 19:36:50.431316 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.431242    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:50.431377 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:50.431328    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:50.433881 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.433852    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jfjzl"]
Jul 28 19:36:50.434005 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.433953    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:50.434061 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:50.434022    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:50.443789 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.443757    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dkh9g"]
Jul 28 19:36:50.443915 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:50.443858    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:50.443984 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:50.443959    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:51.979036 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:51.978870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:51.979362 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:51.978941    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:51.979362 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:51.978959    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:51.979362 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:51.979227    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:51.979362 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:51.979301    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:51.979362 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:51.979106    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:52.230858 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:52.230798    2555 generic.go:356] "Generic (PLEG): container finished" podID="eddbfddc-1f2d-4857-8ad5-698e5fcf4d77" containerID="2f0427c3cce70df77b675d680223015935f18a6558d77b93363ab954efd69b31" exitCode=0
Jul 28 19:36:52.230965 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:52.230875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" event={"ID":"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77","Type":"ContainerDied","Data":"2f0427c3cce70df77b675d680223015935f18a6558d77b93363ab954efd69b31"}
Jul 28 19:36:52.975582 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:52.975512    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:52.975739 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:52.975681    2555 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 19:36:52.976215 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:52.976189    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2569w"
Jul 28 19:36:53.979118 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:53.979086    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:53.979490 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:53.979094    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:53.979490 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:53.979197    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:53.979490 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:53.979192    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jfjzl" podUID="6af25eb8-7d06-493e-8d26-343b128261e1"
Jul 28 19:36:53.979490 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:53.979306    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkh9g" podUID="62542e05-f80b-4490-9213-693a171a19d6"
Jul 28 19:36:53.979490 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:53.979374    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9nkx9" podUID="464229a3-6aff-4f24-ba21-64b65a2182a4"
Jul 28 19:36:55.661263 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.661233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:55.661806 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.661290    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:55.661806 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:55.661386    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:55.661806 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:55.661451    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs podName:62542e05-f80b-4490-9213-693a171a19d6 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.661431675 +0000 UTC m=+66.122190267 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs") pod "network-metrics-daemon-dkh9g" (UID: "62542e05-f80b-4490-9213-693a171a19d6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:36:55.661806 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:55.661394    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:36:55.661806 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:55.661487    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:36:55.661806 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:55.661500    2555 projected.go:196] Error preparing data for projected volume kube-api-access-v2mnz for pod openshift-network-diagnostics/network-check-target-jfjzl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:55.661806 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:55.661542    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz podName:6af25eb8-7d06-493e-8d26-343b128261e1 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.661530619 +0000 UTC m=+66.122289195 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2mnz" (UniqueName: "kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz") pod "network-check-target-jfjzl" (UID: "6af25eb8-7d06-493e-8d26-343b128261e1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:36:55.761881 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.761851    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:55.762036 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:55.761976    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:55.762103 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:55.762037    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret podName:464229a3-6aff-4f24-ba21-64b65a2182a4 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.762019713 +0000 UTC m=+66.222778293 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret") pod "global-pull-secret-syncer-9nkx9" (UID: "464229a3-6aff-4f24-ba21-64b65a2182a4") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:36:55.898481 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.898451    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-173.ec2.internal" event="NodeReady"
Jul 28 19:36:55.898607 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.898567    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 19:36:55.965063 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.964987    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7889b89796-kjvfn"]
Jul 28 19:36:55.981280 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.981257    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:55.984677 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.984232    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 19:36:55.984677 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.984430    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 19:36:55.984874 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.984714    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9t265\""
Jul 28 19:36:55.984874 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.984867    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 19:36:55.985401 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.985382    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tlzp2"]
Jul 28 19:36:55.990335 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:55.990316    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 19:36:56.007042 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.007024    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x7zrk"]
Jul 28 19:36:56.007176 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.007151    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.009788 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.009761    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 19:36:56.009882 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.009863    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pkrrg\""
Jul 28 19:36:56.009963 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.009769    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 19:36:56.025190 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.025167    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7889b89796-kjvfn"]
Jul 28 19:36:56.025289 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.025238    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7zrk"]
Jul 28 19:36:56.025289 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.025258    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tlzp2"]
Jul 28 19:36:56.025401 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.025298    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7zrk"
Jul 28 19:36:56.025401 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.025350    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:36:56.025401 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.025381    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:36:56.025553 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.025307    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:36:56.028213 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.028196    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 19:36:56.028875 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.028719    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 19:36:56.028875 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.028869    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h6gvt\""
Jul 28 19:36:56.029223 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.029200    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 19:36:56.029351 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.029330    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 19:36:56.029492 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.029334    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 19:36:56.029492 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.029383    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jfqx7\""
Jul 28 19:36:56.029492 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.029436    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 19:36:56.029753 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.029736    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-phccc\""
Jul 28 19:36:56.029828 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.029741    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 19:36:56.164388 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164357    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42455482-d5c7-4560-967e-e4ee3568e209-ca-trust-extracted\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.164388 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164385    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-registry-certificates\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.164595 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164404    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.164595 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164485    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-installation-pull-secrets\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.164595 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.164595 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164535    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d160f29-45be-4be6-8ff0-a0837ef292f7-cert\") pod \"ingress-canary-x7zrk\" (UID: \"5d160f29-45be-4be6-8ff0-a0837ef292f7\") " pod="openshift-ingress-canary/ingress-canary-x7zrk"
Jul 28 19:36:56.164595 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164553    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-tmp-dir\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.164595 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164572    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-image-registry-private-configuration\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.164838 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-bound-sa-token\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.164838 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164685    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kncgh\" (UniqueName: \"kubernetes.io/projected/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-kube-api-access-kncgh\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.164838 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164706    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-trusted-ca\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.164838 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164723    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-config-volume\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.164838 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5vr7\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-kube-api-access-s5vr7\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.164838 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.164802    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knntr\" (UniqueName: \"kubernetes.io/projected/5d160f29-45be-4be6-8ff0-a0837ef292f7-kube-api-access-knntr\") pod \"ingress-canary-x7zrk\" (UID: \"5d160f29-45be-4be6-8ff0-a0837ef292f7\") " pod="openshift-ingress-canary/ingress-canary-x7zrk"
Jul 28 19:36:56.266045 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.265987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5vr7\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-kube-api-access-s5vr7\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266045 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266012    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knntr\" (UniqueName: \"kubernetes.io/projected/5d160f29-45be-4be6-8ff0-a0837ef292f7-kube-api-access-knntr\") pod \"ingress-canary-x7zrk\" (UID: \"5d160f29-45be-4be6-8ff0-a0837ef292f7\") " pod="openshift-ingress-canary/ingress-canary-x7zrk"
Jul 28 19:36:56.266184 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42455482-d5c7-4560-967e-e4ee3568e209-ca-trust-extracted\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266184 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-registry-certificates\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266184 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266095    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.266184 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266156    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-installation-pull-secrets\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266184 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.266175    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266202    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d160f29-45be-4be6-8ff0-a0837ef292f7-cert\") pod \"ingress-canary-x7zrk\" (UID: \"5d160f29-45be-4be6-8ff0-a0837ef292f7\") " pod="openshift-ingress-canary/ingress-canary-x7zrk"
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.266237    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls podName:ac2f6cd6-d537-45e7-b8a5-03034fd92b25 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:56.76621951 +0000 UTC m=+35.226978107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls") pod "dns-default-tlzp2" (UID: "ac2f6cd6-d537-45e7-b8a5-03034fd92b25") : secret "dns-default-metrics-tls" not found
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266256    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-tmp-dir\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-image-registry-private-configuration\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-bound-sa-token\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266360    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kncgh\" (UniqueName: \"kubernetes.io/projected/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-kube-api-access-kncgh\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266389    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-trusted-ca\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266414    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-config-volume\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.266843 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266459    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42455482-d5c7-4560-967e-e4ee3568e209-ca-trust-extracted\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266843 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266628    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-tmp-dir\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.266843 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-registry-certificates\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.266843 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.266781    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:36:56.266843 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.266798    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7889b89796-kjvfn: secret "image-registry-tls" not found
Jul 28 19:36:56.267087 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.266861    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls podName:42455482-d5c7-4560-967e-e4ee3568e209 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:56.766841078 +0000 UTC m=+35.227599661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls") pod "image-registry-7889b89796-kjvfn" (UID: "42455482-d5c7-4560-967e-e4ee3568e209") : secret "image-registry-tls" not found
Jul 28 19:36:56.267087 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.266918    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-config-volume\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.267570 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.267554    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-trusted-ca\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.270146 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.270125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-installation-pull-secrets\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.270229 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.270131    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d160f29-45be-4be6-8ff0-a0837ef292f7-cert\") pod \"ingress-canary-x7zrk\" (UID: \"5d160f29-45be-4be6-8ff0-a0837ef292f7\") " pod="openshift-ingress-canary/ingress-canary-x7zrk"
Jul 28 19:36:56.270229 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.270172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-image-registry-private-configuration\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.277295 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.277277    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kncgh\" (UniqueName: \"kubernetes.io/projected/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-kube-api-access-kncgh\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.277802 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.277785    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5vr7\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-kube-api-access-s5vr7\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.277855 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.277818    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-bound-sa-token\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.277940 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.277925    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knntr\" (UniqueName: \"kubernetes.io/projected/5d160f29-45be-4be6-8ff0-a0837ef292f7-kube-api-access-knntr\") pod \"ingress-canary-x7zrk\" (UID: \"5d160f29-45be-4be6-8ff0-a0837ef292f7\") " pod="openshift-ingress-canary/ingress-canary-x7zrk"
Jul 28 19:36:56.336059 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.336039    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x7zrk"
Jul 28 19:36:56.711446 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.711260    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x7zrk"]
Jul 28 19:36:56.715232 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:36:56.715206    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d160f29_45be_4be6_8ff0_a0837ef292f7.slice/crio-0efb791c91a5f575711b68ba9f140d67003da13010fced57f0f29307ad4740ba WatchSource:0}: Error finding container 0efb791c91a5f575711b68ba9f140d67003da13010fced57f0f29307ad4740ba: Status 404 returned error can't find the container with id 0efb791c91a5f575711b68ba9f140d67003da13010fced57f0f29307ad4740ba
Jul 28 19:36:56.768732 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.768698    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:56.768882 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:56.768743    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:56.768882 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.768864    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:36:56.769025 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.768904    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:36:56.769025 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.768917    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7889b89796-kjvfn: secret "image-registry-tls" not found
Jul 28 19:36:56.769025 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.768957    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls podName:ac2f6cd6-d537-45e7-b8a5-03034fd92b25 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:57.768936071 +0000 UTC m=+36.229694650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls") pod "dns-default-tlzp2" (UID: "ac2f6cd6-d537-45e7-b8a5-03034fd92b25") : secret "dns-default-metrics-tls" not found
Jul 28 19:36:56.769025 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:56.768977    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls podName:42455482-d5c7-4560-967e-e4ee3568e209 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:57.768968257 +0000 UTC m=+36.229726837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls") pod "image-registry-7889b89796-kjvfn" (UID: "42455482-d5c7-4560-967e-e4ee3568e209") : secret "image-registry-tls" not found
Jul 28 19:36:57.242374 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:57.242295    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7zrk" event={"ID":"5d160f29-45be-4be6-8ff0-a0837ef292f7","Type":"ContainerStarted","Data":"0efb791c91a5f575711b68ba9f140d67003da13010fced57f0f29307ad4740ba"}
Jul 28 19:36:57.245556 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:57.245524    2555 generic.go:356] "Generic (PLEG): container finished" podID="eddbfddc-1f2d-4857-8ad5-698e5fcf4d77" containerID="903568e5411b194cef09e6de3b237feabf8fed4a90817ed424a2a590d3e9b0fb" exitCode=0
Jul 28 19:36:57.245671 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:57.245580    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" event={"ID":"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77","Type":"ContainerDied","Data":"903568e5411b194cef09e6de3b237feabf8fed4a90817ed424a2a590d3e9b0fb"}
Jul 28 19:36:57.776481 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:57.776448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:57.776916 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:57.776503    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:57.776916 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:57.776618    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:36:57.776916 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:57.776625    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:36:57.776916 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:57.776699    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls podName:ac2f6cd6-d537-45e7-b8a5-03034fd92b25 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:59.776677351 +0000 UTC m=+38.237435945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls") pod "dns-default-tlzp2" (UID: "ac2f6cd6-d537-45e7-b8a5-03034fd92b25") : secret "dns-default-metrics-tls" not found
Jul 28 19:36:57.776916 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:57.776632    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7889b89796-kjvfn: secret "image-registry-tls" not found
Jul 28 19:36:57.776916 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:57.776778    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls podName:42455482-d5c7-4560-967e-e4ee3568e209 nodeName:}" failed. No retries permitted until 2026-07-28 19:36:59.77676159 +0000 UTC m=+38.237520168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls") pod "image-registry-7889b89796-kjvfn" (UID: "42455482-d5c7-4560-967e-e4ee3568e209") : secret "image-registry-tls" not found
Jul 28 19:36:58.251573 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:58.251479    2555 generic.go:356] "Generic (PLEG): container finished" podID="eddbfddc-1f2d-4857-8ad5-698e5fcf4d77" containerID="4f8c6caab42629d346a5ee3c29992ffe280a4b26ce76bbbaff0f8db209073949" exitCode=0
Jul 28 19:36:58.251573 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:58.251558    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" event={"ID":"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77","Type":"ContainerDied","Data":"4f8c6caab42629d346a5ee3c29992ffe280a4b26ce76bbbaff0f8db209073949"}
Jul 28 19:36:59.258062 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:59.257833    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" event={"ID":"eddbfddc-1f2d-4857-8ad5-698e5fcf4d77","Type":"ContainerStarted","Data":"a8542390ee347460371c3975bd77b77b3e3723e70416722fb24c4f40066b2d78"}
Jul 28 19:36:59.259257 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:59.259234    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x7zrk" event={"ID":"5d160f29-45be-4be6-8ff0-a0837ef292f7","Type":"ContainerStarted","Data":"e15ae26e479f33a5bfa410d23981591e735a9d2fe48d211b8aa200b42baa5e2a"}
Jul 28 19:36:59.283222 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:59.283168    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mj8cx" podStartSLOduration=8.473278657 podStartE2EDuration="37.28314965s" totalImagesPullingTime="28.809870993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:59.280851457 +0000 UTC m=+37.741610055" watchObservedRunningTime="2026-07-28 19:36:59.28314965 +0000 UTC m=+37.743908250"
Jul 28 19:36:59.315309 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:59.315216    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x7zrk" podStartSLOduration=1.905862971 podStartE2EDuration="4.315197921s" totalImagesPullingTime="2.40933495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:36:59.314396414 +0000 UTC m=+37.775155014" watchObservedRunningTime="2026-07-28 19:36:59.315197921 +0000 UTC m=+37.775956521"
Jul 28 19:36:59.466817 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:59.466744    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:36:59.562780 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:59.562755    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:36:59.790937 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:59.790848    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:36:59.790937 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:36:59.790916    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:36:59.791138 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:59.791000    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:36:59.791138 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:59.791011    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7889b89796-kjvfn: secret "image-registry-tls" not found
Jul 28 19:36:59.791138 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:59.791019    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:36:59.791138 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:59.791051    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls podName:42455482-d5c7-4560-967e-e4ee3568e209 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.791038316 +0000 UTC m=+42.251796893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls") pod "image-registry-7889b89796-kjvfn" (UID: "42455482-d5c7-4560-967e-e4ee3568e209") : secret "image-registry-tls" not found
Jul 28 19:36:59.791138 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:36:59.791108    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls podName:ac2f6cd6-d537-45e7-b8a5-03034fd92b25 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.791091356 +0000 UTC m=+42.251849937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls") pod "dns-default-tlzp2" (UID: "ac2f6cd6-d537-45e7-b8a5-03034fd92b25") : secret "dns-default-metrics-tls" not found
Jul 28 19:37:00.060967 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:00.060906    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:00.157503 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:00.157471    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:00.361738 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:00.361677    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:00.591999 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:00.591962    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:00.652357 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:00.652289    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:00.704499 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:00.704469    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:01.499712 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.499676    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:01.589426 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.589371    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"]
Jul 28 19:37:01.589621 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.589600    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:01.591303 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.591286    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:01.593992 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.593972    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 19:37:01.594107 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.594023    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 19:37:01.594107 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.594097    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ltwr9\""
Jul 28 19:37:01.602553 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.602531    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"]
Jul 28 19:37:01.704905 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.704870    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:01.705023 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.705001    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea2f8539-0a89-4894-8bd6-73822815662e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:01.725735 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.725706    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:01.806212 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.806160    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea2f8539-0a89-4894-8bd6-73822815662e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:01.806276 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.806218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:01.806334 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:01.806320    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:37:01.806467 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:01.806381    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert podName:ea2f8539-0a89-4894-8bd6-73822815662e nodeName:}" failed. No retries permitted until 2026-07-28 19:37:02.306368216 +0000 UTC m=+40.767126793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnll5" (UID: "ea2f8539-0a89-4894-8bd6-73822815662e") : secret "networking-console-plugin-cert" not found
Jul 28 19:37:01.806855 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.806836    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ea2f8539-0a89-4894-8bd6-73822815662e-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:01.825817 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.825795    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:01.879351 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.879326    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:01.982872 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:01.982852    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:02.052617 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:02.052597    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:02.309038 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:02.309013    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:02.309153 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:02.309140    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:37:02.309207 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:02.309193    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert podName:ea2f8539-0a89-4894-8bd6-73822815662e nodeName:}" failed. No retries permitted until 2026-07-28 19:37:03.309179143 +0000 UTC m=+41.769937719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnll5" (UID: "ea2f8539-0a89-4894-8bd6-73822815662e") : secret "networking-console-plugin-cert" not found
Jul 28 19:37:03.317961 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:03.317927    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:03.318328 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:03.318069    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:37:03.318328 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:03.318127    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert podName:ea2f8539-0a89-4894-8bd6-73822815662e nodeName:}" failed. No retries permitted until 2026-07-28 19:37:05.318111482 +0000 UTC m=+43.778870059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnll5" (UID: "ea2f8539-0a89-4894-8bd6-73822815662e") : secret "networking-console-plugin-cert" not found
Jul 28 19:37:03.820963 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:03.820932    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:37:03.821087 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:03.820978    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:37:03.821121 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:03.821082    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:37:03.821121 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:03.821105    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:37:03.821121 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:03.821116    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7889b89796-kjvfn: secret "image-registry-tls" not found
Jul 28 19:37:03.821215 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:03.821180    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls podName:42455482-d5c7-4560-967e-e4ee3568e209 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:11.821162738 +0000 UTC m=+50.281921330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls") pod "image-registry-7889b89796-kjvfn" (UID: "42455482-d5c7-4560-967e-e4ee3568e209") : secret "image-registry-tls" not found
Jul 28 19:37:03.821215 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:03.821193    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls podName:ac2f6cd6-d537-45e7-b8a5-03034fd92b25 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:11.821187386 +0000 UTC m=+50.281945963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls") pod "dns-default-tlzp2" (UID: "ac2f6cd6-d537-45e7-b8a5-03034fd92b25") : secret "dns-default-metrics-tls" not found
Jul 28 19:37:05.263059 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:05.263025    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:05.331241 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:05.331211    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:05.331419 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:05.331380    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:37:05.331487 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:05.331467    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert podName:ea2f8539-0a89-4894-8bd6-73822815662e nodeName:}" failed. No retries permitted until 2026-07-28 19:37:09.331444698 +0000 UTC m=+47.792203283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnll5" (UID: "ea2f8539-0a89-4894-8bd6-73822815662e") : secret "networking-console-plugin-cert" not found
Jul 28 19:37:05.364953 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:05.364925    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:06.517314 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.517284    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:06.629579 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.629537    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:06.750405 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.750373    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gq6td"]
Jul 28 19:37:06.783779 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.783709    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:06.783965 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.783949    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gq6td"]
Jul 28 19:37:06.784071 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.784057    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:06.786463 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.786442    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 19:37:06.787436 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.787417    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 19:37:06.787564 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.787454    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 19:37:06.787564 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.787468    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cdlcv\""
Jul 28 19:37:06.787564 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.787470    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 19:37:06.857295 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.857269    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:06.944575 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.944546    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-crio-socket\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:06.944575 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.944578    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:06.944761 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.944596    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rwg\" (UniqueName: \"kubernetes.io/projected/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-api-access-56rwg\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:06.944761 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.944710    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:06.944824 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.944762    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-data-volume\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:06.985145 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.985100    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:06.988683 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:06.988658    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj"]
Jul 28 19:37:07.005695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.005674    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj"]
Jul 28 19:37:07.005783 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.005760    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj"
Jul 28 19:37:07.008848 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.008824    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 19:37:07.008952 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.008844    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 19:37:07.009066 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.009049    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xklwp\""
Jul 28 19:37:07.045208 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.045190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.045291 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.045232    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-data-volume\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.045334 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:07.045304    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:37:07.045367 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.045346    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-crio-socket\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.045367 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:07.045360    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:07.5453457 +0000 UTC m=+46.006104318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : configmap "kube-rbac-proxy" not found
Jul 28 19:37:07.045444 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.045384    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.045444 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.045411    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rwg\" (UniqueName: \"kubernetes.io/projected/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-api-access-56rwg\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.045538 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.045466    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-crio-socket\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.045538 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:07.045525    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:07.045641 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:07.045583    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:07.545564558 +0000 UTC m=+46.006323152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:07.045697 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.045678    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-data-volume\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.055417 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.055394    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rwg\" (UniqueName: \"kubernetes.io/projected/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-api-access-56rwg\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.057312 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.057288    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:07.127345 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.127326    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:07.145871 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.145849    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb44h\" (UniqueName: \"kubernetes.io/projected/8ed193d5-940f-4ec4-8c19-53f4ecef8e77-kube-api-access-sb44h\") pod \"migrator-6865c9967-w5wkj\" (UID: \"8ed193d5-940f-4ec4-8c19-53f4ecef8e77\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj"
Jul 28 19:37:07.211632 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.211585    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:07.246782 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.246761    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb44h\" (UniqueName: \"kubernetes.io/projected/8ed193d5-940f-4ec4-8c19-53f4ecef8e77-kube-api-access-sb44h\") pod \"migrator-6865c9967-w5wkj\" (UID: \"8ed193d5-940f-4ec4-8c19-53f4ecef8e77\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj"
Jul 28 19:37:07.256064 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.256035    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb44h\" (UniqueName: \"kubernetes.io/projected/8ed193d5-940f-4ec4-8c19-53f4ecef8e77-kube-api-access-sb44h\") pod \"migrator-6865c9967-w5wkj\" (UID: \"8ed193d5-940f-4ec4-8c19-53f4ecef8e77\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj"
Jul 28 19:37:07.309482 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.309392    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:07.314306 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.314291    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj"
Jul 28 19:37:07.407071 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.407047    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:07.449350 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.449320    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj"]
Jul 28 19:37:07.467966 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:07.467944    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed193d5_940f_4ec4_8c19_53f4ecef8e77.slice/crio-9ed01507a581ae88fdf9573566dd367a5e4c942445297816ace29d60caaa1ae5 WatchSource:0}: Error finding container 9ed01507a581ae88fdf9573566dd367a5e4c942445297816ace29d60caaa1ae5: Status 404 returned error can't find the container with id 9ed01507a581ae88fdf9573566dd367a5e4c942445297816ace29d60caaa1ae5
Jul 28 19:37:07.549451 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.549430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.549742 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:07.549485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:07.549742 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:07.549569    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:37:07.549742 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:07.549584    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:07.549742 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:07.549614    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:08.54960063 +0000 UTC m=+47.010359207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : configmap "kube-rbac-proxy" not found
Jul 28 19:37:07.549742 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:07.549638    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:08.549625625 +0000 UTC m=+47.010384206 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:08.135524 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.135493    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-67xlq_666e0bbd-3122-4414-b121-88468187b07c/dns-node-resolver/0.log"
Jul 28 19:37:08.243035 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.243005    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:08.280276 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.280243    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj" event={"ID":"8ed193d5-940f-4ec4-8c19-53f4ecef8e77","Type":"ContainerStarted","Data":"9ed01507a581ae88fdf9573566dd367a5e4c942445297816ace29d60caaa1ae5"}
Jul 28 19:37:08.297754 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.297714    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:08.556412 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.556375    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:08.556925 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.556487    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:08.556925 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:08.556555    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:08.556925 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:08.556584    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:37:08.556925 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:08.556639    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:10.556616066 +0000 UTC m=+49.017374649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:08.556925 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:08.556660    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:10.556650778 +0000 UTC m=+49.017409355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : configmap "kube-rbac-proxy" not found
Jul 28 19:37:08.814656 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.814576    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5xxz8"]
Jul 28 19:37:08.833069 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.833035    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5xxz8"]
Jul 28 19:37:08.833234 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.833153    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:08.835903 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.835853    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 19:37:08.835903 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.835882    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 19:37:08.836767 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.836749    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9dbss\""
Jul 28 19:37:08.836882 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.836857    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 19:37:08.836981 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.836963    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 19:37:08.859644 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.859622    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:08.959543 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.959513    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-signing-key\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:08.959543 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.959545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-signing-cabundle\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:08.959719 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:08.959568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh8q\" (UniqueName: \"kubernetes.io/projected/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-kube-api-access-zmh8q\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:09.060117 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.060089    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-signing-key\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:09.060234 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.060122    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-signing-cabundle\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:09.060234 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.060145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh8q\" (UniqueName: \"kubernetes.io/projected/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-kube-api-access-zmh8q\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:09.060681 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.060660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-signing-cabundle\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:09.062431 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.062409    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-signing-key\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:09.068581 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.068527    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh8q\" (UniqueName: \"kubernetes.io/projected/0b1f7e9a-2663-4051-9a08-9d8e5f784d42-kube-api-access-zmh8q\") pod \"service-ca-658d7b7c78-5xxz8\" (UID: \"0b1f7e9a-2663-4051-9a08-9d8e5f784d42\") " pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:09.144538 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.144514    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8"
Jul 28 19:37:09.263202 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.263157    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:09.362748 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.362723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:09.362874 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:09.362836    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:37:09.362979 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:09.362965    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert podName:ea2f8539-0a89-4894-8bd6-73822815662e nodeName:}" failed. No retries permitted until 2026-07-28 19:37:17.362875157 +0000 UTC m=+55.823633736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnll5" (UID: "ea2f8539-0a89-4894-8bd6-73822815662e") : secret "networking-console-plugin-cert" not found
Jul 28 19:37:09.415680 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.415652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-5xxz8"]
Jul 28 19:37:09.418344 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:09.418311    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1f7e9a_2663_4051_9a08_9d8e5f784d42.slice/crio-87d5ec8a06d1cdc4a4d6aa0d3320f70c530576db25cc2cb215cbb57023fa5840 WatchSource:0}: Error finding container 87d5ec8a06d1cdc4a4d6aa0d3320f70c530576db25cc2cb215cbb57023fa5840: Status 404 returned error can't find the container with id 87d5ec8a06d1cdc4a4d6aa0d3320f70c530576db25cc2cb215cbb57023fa5840
Jul 28 19:37:09.537373 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.537355    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pljjh_674936af-b442-43be-b18d-2677522519c3/node-ca/0.log"
Jul 28 19:37:09.829192 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:09.829161    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:10.138603 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:10.138528    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7zrk_5d160f29-45be-4be6-8ff0-a0837ef292f7/serve-healthcheck-canary/0.log"
Jul 28 19:37:10.287136 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:10.287102    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj" event={"ID":"8ed193d5-940f-4ec4-8c19-53f4ecef8e77","Type":"ContainerStarted","Data":"d322f3b7ebaaca62a87e4789e5ebb2b0133130c923ea6e3acb8fe02cd04592f4"}
Jul 28 19:37:10.287311 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:10.287160    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj" event={"ID":"8ed193d5-940f-4ec4-8c19-53f4ecef8e77","Type":"ContainerStarted","Data":"c021de99d8ac3b21e21222fa6a2e428571d58cc0fc9029001d548e51fa7fa4ef"}
Jul 28 19:37:10.288461 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:10.288426    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8" event={"ID":"0b1f7e9a-2663-4051-9a08-9d8e5f784d42","Type":"ContainerStarted","Data":"87d5ec8a06d1cdc4a4d6aa0d3320f70c530576db25cc2cb215cbb57023fa5840"}
Jul 28 19:37:10.305020 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:10.304966    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-w5wkj" podStartSLOduration=2.497821262 podStartE2EDuration="4.30494732s" totalImagesPullingTime="1.807126058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:10.30461443 +0000 UTC m=+48.765373030" watchObservedRunningTime="2026-07-28 19:37:10.30494732 +0000 UTC m=+48.765705921"
Jul 28 19:37:10.572022 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:10.571991    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:10.572203 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:10.572094    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:10.572203 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:10.572119    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:37:10.572203 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:10.572184    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:14.572168563 +0000 UTC m=+53.032927142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : configmap "kube-rbac-proxy" not found
Jul 28 19:37:10.572371 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:10.572234    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:10.572371 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:10.572293    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:14.572276028 +0000 UTC m=+53.033034630 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:11.884024 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:11.883997    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:37:11.884377 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:11.884045    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:37:11.884377 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:11.884144    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:37:11.884377 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:11.884181    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:37:11.884377 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:11.884191    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7889b89796-kjvfn: secret "image-registry-tls" not found
Jul 28 19:37:11.884377 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:11.884202    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls podName:ac2f6cd6-d537-45e7-b8a5-03034fd92b25 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.884183619 +0000 UTC m=+66.344942199 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls") pod "dns-default-tlzp2" (UID: "ac2f6cd6-d537-45e7-b8a5-03034fd92b25") : secret "dns-default-metrics-tls" not found
Jul 28 19:37:11.884377 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:11.884224    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls podName:42455482-d5c7-4560-967e-e4ee3568e209 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:27.884213349 +0000 UTC m=+66.344971930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls") pod "image-registry-7889b89796-kjvfn" (UID: "42455482-d5c7-4560-967e-e4ee3568e209") : secret "image-registry-tls" not found
Jul 28 19:37:12.050399 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:12.050375    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:12.295122 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:12.295086    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8" event={"ID":"0b1f7e9a-2663-4051-9a08-9d8e5f784d42","Type":"ContainerStarted","Data":"61b503584aebebec23288ba87a17dc2dd6688e3fd75fe18c082e1570c2817bfd"}
Jul 28 19:37:12.314006 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:12.313920    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-5xxz8" podStartSLOduration=1.984669713 podStartE2EDuration="4.313905974s" totalImagesPullingTime="2.329236261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:12.313115774 +0000 UTC m=+50.773874374" watchObservedRunningTime="2026-07-28 19:37:12.313905974 +0000 UTC m=+50.774664564"
Jul 28 19:37:12.556041 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:12.556006    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:12.903447 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:12.903412    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:13.004727 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:13.004687    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:13.107069 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:13.107040    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:13.181323 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:13.181245    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:13.246189 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:13.246162    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:14.206443 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:14.206416    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:14.604703 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:14.604670    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:14.604912 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:14.604731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:14.604912 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:14.604822    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:14.604912 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:14.604902    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls podName:2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5 nodeName:}" failed. No retries permitted until 2026-07-28 19:37:22.604867485 +0000 UTC m=+61.065626073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gq6td" (UID: "2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:37:14.605316 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:14.605299    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:14.608629 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:14.608611    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:14.802408 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:14.802377    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:15.005292 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:15.005227    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:15.404864 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:15.404832    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:15.886852 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:15.886828    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:16.001787 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:16.001766    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:16.208033 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:16.207976    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:16.625156 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:16.625119    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:16.806916 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:16.806881    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:16.863200 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:16.863178    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:17.200162 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:17.200134    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:17.413197 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:17.413175    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:17.428326 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:17.428307    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:17.428411 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:17.428400    2555 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:37:17.428458 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:37:17.428449    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert podName:ea2f8539-0a89-4894-8bd6-73822815662e nodeName:}" failed. No retries permitted until 2026-07-28 19:37:33.428435299 +0000 UTC m=+71.889193876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnll5" (UID: "ea2f8539-0a89-4894-8bd6-73822815662e") : secret "networking-console-plugin-cert" not found
Jul 28 19:37:17.516013 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:17.515967    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:18.003683 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:18.003659    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:18.407390 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:18.407366    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:18.808177 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:18.808152    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:19.125437 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:19.125391    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:19.227577 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:19.227557    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:19.407012 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:19.406961    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:19.801066 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:19.801033    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:20.260473 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:20.260422    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:20.361056 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:20.361029    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:20.462474 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:20.462453    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:20.799690 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:20.799666    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:21.204609 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:21.204542    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:21.399542 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:21.399520    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:22.000010 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:21.999986    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:22.249874 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:22.249854    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dx4p9"
Jul 28 19:37:22.404942 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:22.404913    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:22.666646 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:22.666579    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:22.682619 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:22.682596    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gq6td\" (UID: \"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5\") " pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:22.695423 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:22.695407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cdlcv\""
Jul 28 19:37:22.703334 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:22.703309    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gq6td"
Jul 28 19:37:22.826223 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:22.826200    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:22.845428 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:22.845404    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gq6td"]
Jul 28 19:37:22.849116 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:22.849084    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc74456_ac91_4a6b_8f18_c5bb3dd7ecc5.slice/crio-4d019e859e1ddf3ae78b9fdc19791e367396095689170f5d22c8343bce977983 WatchSource:0}: Error finding container 4d019e859e1ddf3ae78b9fdc19791e367396095689170f5d22c8343bce977983: Status 404 returned error can't find the container with id 4d019e859e1ddf3ae78b9fdc19791e367396095689170f5d22c8343bce977983
Jul 28 19:37:23.203237 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:23.203187    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:23.321252 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:23.321228    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gq6td" event={"ID":"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5","Type":"ContainerStarted","Data":"4d019e859e1ddf3ae78b9fdc19791e367396095689170f5d22c8343bce977983"}
Jul 28 19:37:23.402471 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:23.402450    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:23.609944 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:23.609908    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:25.497270 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:25.497235    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:27.704786 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.704747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:37:27.705207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.704803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:37:27.707430 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.707407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 19:37:27.707502 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.707409    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 19:37:27.717531 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.717511    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62542e05-f80b-4490-9213-693a171a19d6-metrics-certs\") pod \"network-metrics-daemon-dkh9g\" (UID: \"62542e05-f80b-4490-9213-693a171a19d6\") " pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:37:27.717969 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.717954    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 19:37:27.728074 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.728053    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2mnz\" (UniqueName: \"kubernetes.io/projected/6af25eb8-7d06-493e-8d26-343b128261e1-kube-api-access-v2mnz\") pod \"network-check-target-jfjzl\" (UID: \"6af25eb8-7d06-493e-8d26-343b128261e1\") " pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:37:27.805857 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.805838    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:37:27.808219 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.808204    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 19:37:27.817926 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.817903    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/464229a3-6aff-4f24-ba21-64b65a2182a4-original-pull-secret\") pod \"global-pull-secret-syncer-9nkx9\" (UID: \"464229a3-6aff-4f24-ba21-64b65a2182a4\") " pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:37:27.845453 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.845428    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h6gvt\""
Jul 28 19:37:27.849666 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.849652    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9nkx9"
Jul 28 19:37:27.853355 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.853338    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkh9g"
Jul 28 19:37:27.857857 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.857839    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jfqx7\""
Jul 28 19:37:27.865997 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.865978    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:37:27.908973 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.907251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:37:27.908973 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.907393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:37:27.911508 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.910340    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ac2f6cd6-d537-45e7-b8a5-03034fd92b25-metrics-tls\") pod \"dns-default-tlzp2\" (UID: \"ac2f6cd6-d537-45e7-b8a5-03034fd92b25\") " pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:37:27.911508 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.911240    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"image-registry-7889b89796-kjvfn\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") " pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:37:27.984077 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:27.984057    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9nkx9"]
Jul 28 19:37:27.985204 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:27.985176    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464229a3_6aff_4f24_ba21_64b65a2182a4.slice/crio-481feec6c8585859d773d3eff7920f42c473bf4a08123ed92c6b44d72443167b WatchSource:0}: Error finding container 481feec6c8585859d773d3eff7920f42c473bf4a08123ed92c6b44d72443167b: Status 404 returned error can't find the container with id 481feec6c8585859d773d3eff7920f42c473bf4a08123ed92c6b44d72443167b
Jul 28 19:37:28.001283 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.001260    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dkh9g"]
Jul 28 19:37:28.003689 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:28.003667    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62542e05_f80b_4490_9213_693a171a19d6.slice/crio-66d24db1ad4eadad18c9655e74679a48e4827f2afe536eb260d8ec3f77fee167 WatchSource:0}: Error finding container 66d24db1ad4eadad18c9655e74679a48e4827f2afe536eb260d8ec3f77fee167: Status 404 returned error can't find the container with id 66d24db1ad4eadad18c9655e74679a48e4827f2afe536eb260d8ec3f77fee167
Jul 28 19:37:28.016720 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.016698    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jfjzl"]
Jul 28 19:37:28.019320 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:28.019299    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af25eb8_7d06_493e_8d26_343b128261e1.slice/crio-1d3658b1006ae50ef738b9f54f99b3c601eaaeed8e68e8cd28f2e56a1d3d4e7b WatchSource:0}: Error finding container 1d3658b1006ae50ef738b9f54f99b3c601eaaeed8e68e8cd28f2e56a1d3d4e7b: Status 404 returned error can't find the container with id 1d3658b1006ae50ef738b9f54f99b3c601eaaeed8e68e8cd28f2e56a1d3d4e7b
Jul 28 19:37:28.095008 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.094991    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9t265\""
Jul 28 19:37:28.103310 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.103292    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:37:28.118899 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.118859    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pkrrg\""
Jul 28 19:37:28.127309 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.127287    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:37:28.234187 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.234128    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7889b89796-kjvfn"]
Jul 28 19:37:28.236778 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:28.236755    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42455482_d5c7_4560_967e_e4ee3568e209.slice/crio-cf6f0d55571754adc0f1eb27d26ee3dbbbefdf8ea2da59cc8934fa204820305b WatchSource:0}: Error finding container cf6f0d55571754adc0f1eb27d26ee3dbbbefdf8ea2da59cc8934fa204820305b: Status 404 returned error can't find the container with id cf6f0d55571754adc0f1eb27d26ee3dbbbefdf8ea2da59cc8934fa204820305b
Jul 28 19:37:28.255298 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.255276    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tlzp2"]
Jul 28 19:37:28.264783 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:28.264757    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2f6cd6_d537_45e7_b8a5_03034fd92b25.slice/crio-299869326c77ea503559834d2f6e2525df7a4d79088e0f0df16011bf0a4cf8c2 WatchSource:0}: Error finding container 299869326c77ea503559834d2f6e2525df7a4d79088e0f0df16011bf0a4cf8c2: Status 404 returned error can't find the container with id 299869326c77ea503559834d2f6e2525df7a4d79088e0f0df16011bf0a4cf8c2
Jul 28 19:37:28.336476 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.336446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7889b89796-kjvfn" event={"ID":"42455482-d5c7-4560-967e-e4ee3568e209","Type":"ContainerStarted","Data":"3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563"}
Jul 28 19:37:28.336667 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.336499    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7889b89796-kjvfn" event={"ID":"42455482-d5c7-4560-967e-e4ee3568e209","Type":"ContainerStarted","Data":"cf6f0d55571754adc0f1eb27d26ee3dbbbefdf8ea2da59cc8934fa204820305b"}
Jul 28 19:37:28.336667 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.336556    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:37:28.337936 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.337915    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gq6td" event={"ID":"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5","Type":"ContainerStarted","Data":"12e3ef7705d4b99c0c40d33e4cf77aaf5e75d8b2140f24f4320eb8d46eae0c45"}
Jul 28 19:37:28.338956 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.338932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlzp2" event={"ID":"ac2f6cd6-d537-45e7-b8a5-03034fd92b25","Type":"ContainerStarted","Data":"299869326c77ea503559834d2f6e2525df7a4d79088e0f0df16011bf0a4cf8c2"}
Jul 28 19:37:28.340025 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.340002    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9nkx9" event={"ID":"464229a3-6aff-4f24-ba21-64b65a2182a4","Type":"ContainerStarted","Data":"481feec6c8585859d773d3eff7920f42c473bf4a08123ed92c6b44d72443167b"}
Jul 28 19:37:28.341312 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.341171    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jfjzl" event={"ID":"6af25eb8-7d06-493e-8d26-343b128261e1","Type":"ContainerStarted","Data":"1d3658b1006ae50ef738b9f54f99b3c601eaaeed8e68e8cd28f2e56a1d3d4e7b"}
Jul 28 19:37:28.342374 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.342357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkh9g" event={"ID":"62542e05-f80b-4490-9213-693a171a19d6","Type":"ContainerStarted","Data":"66d24db1ad4eadad18c9655e74679a48e4827f2afe536eb260d8ec3f77fee167"}
Jul 28 19:37:28.357999 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:28.357960    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7889b89796-kjvfn" podStartSLOduration=56.357944262 podStartE2EDuration="56.357944262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:28.357178858 +0000 UTC m=+66.817937458" watchObservedRunningTime="2026-07-28 19:37:28.357944262 +0000 UTC m=+66.818702862"
Jul 28 19:37:29.501344 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.500466    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"]
Jul 28 19:37:29.520690 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.520659    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"]
Jul 28 19:37:29.521348 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.521267    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.524791 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.524685    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 19:37:29.525647 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.525622    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 19:37:29.526224 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.525842    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 19:37:29.526224 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.526070    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 19:37:29.540990 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.540967    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"]
Jul 28 19:37:29.541085 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.541013    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"]
Jul 28 19:37:29.541839 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.541444    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"
Jul 28 19:37:29.544769 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.544560    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 28 19:37:29.544769 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.544628    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wfggn\""
Jul 28 19:37:29.557363 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.557343    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"]
Jul 28 19:37:29.557448 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.557376    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"]
Jul 28 19:37:29.557512 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.557466    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.560252 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.560230    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 19:37:29.560490 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.560447    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 19:37:29.560663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.560646    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 19:37:29.560789 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.560655    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 19:37:29.621685 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.621657    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.621798 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.621699    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-ca\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.621798 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.621730    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1dc6dad-e3e3-4200-898a-3dfa57fc10ea-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6\" (UID: \"f1dc6dad-e3e3-4200-898a-3dfa57fc10ea\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"
Jul 28 19:37:29.621798 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.621764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-hub\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.621798 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.621787    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.622075 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.621811    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwflg\" (UniqueName: \"kubernetes.io/projected/86f12d38-15f5-4721-a27a-5f2935352c70-kube-api-access-vwflg\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.622075 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.621902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.622075 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.621951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxhw\" (UniqueName: \"kubernetes.io/projected/f1dc6dad-e3e3-4200-898a-3dfa57fc10ea-kube-api-access-qnxhw\") pod \"managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6\" (UID: \"f1dc6dad-e3e3-4200-898a-3dfa57fc10ea\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"
Jul 28 19:37:29.622075 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.622013    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrmr\" (UniqueName: \"kubernetes.io/projected/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-kube-api-access-snrmr\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.622075 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.622050    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-tmp\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.622075 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.622073    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f12d38-15f5-4721-a27a-5f2935352c70-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.703484 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.703398    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722372    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722412    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-ca\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1dc6dad-e3e3-4200-898a-3dfa57fc10ea-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6\" (UID: \"f1dc6dad-e3e3-4200-898a-3dfa57fc10ea\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722481    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-hub\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722504    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722528    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwflg\" (UniqueName: \"kubernetes.io/projected/86f12d38-15f5-4721-a27a-5f2935352c70-kube-api-access-vwflg\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722558    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxhw\" (UniqueName: \"kubernetes.io/projected/f1dc6dad-e3e3-4200-898a-3dfa57fc10ea-kube-api-access-qnxhw\") pod \"managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6\" (UID: \"f1dc6dad-e3e3-4200-898a-3dfa57fc10ea\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722648    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrmr\" (UniqueName: \"kubernetes.io/projected/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-kube-api-access-snrmr\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.722695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.722686    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-tmp\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.723264 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.723041    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-tmp\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.724834 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.723898    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f12d38-15f5-4721-a27a-5f2935352c70-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.724834 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.724768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/86f12d38-15f5-4721-a27a-5f2935352c70-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.729273 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.729221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-ca\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.729373 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.729275    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.733918 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.733560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-hub\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.734133 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.734023    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1dc6dad-e3e3-4200-898a-3dfa57fc10ea-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6\" (UID: \"f1dc6dad-e3e3-4200-898a-3dfa57fc10ea\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"
Jul 28 19:37:29.734133 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.734109    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86f12d38-15f5-4721-a27a-5f2935352c70-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.734618 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.734597    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-klusterlet-config\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.747549 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.747505    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwflg\" (UniqueName: \"kubernetes.io/projected/86f12d38-15f5-4721-a27a-5f2935352c70-kube-api-access-vwflg\") pod \"cluster-proxy-proxy-agent-8487f78744-9dbhj\" (UID: \"86f12d38-15f5-4721-a27a-5f2935352c70\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.748592 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.748552    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrmr\" (UniqueName: \"kubernetes.io/projected/cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37-kube-api-access-snrmr\") pod \"klusterlet-addon-workmgr-5fd95867b4-ng2j5\" (UID: \"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.751393 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.751322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxhw\" (UniqueName: \"kubernetes.io/projected/f1dc6dad-e3e3-4200-898a-3dfa57fc10ea-kube-api-access-qnxhw\") pod \"managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6\" (UID: \"f1dc6dad-e3e3-4200-898a-3dfa57fc10ea\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"
Jul 28 19:37:29.810166 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.810133    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:29.836140 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.836116    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:29.867420 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.867379    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"
Jul 28 19:37:29.879548 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.878814    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"
Jul 28 19:37:29.905111 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:29.905072    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:30.064051 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.063983    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"]
Jul 28 19:37:30.066059 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:30.065981    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabaeed5_a3dd_440e_a8b0_df4ea3cc9c37.slice/crio-673fef636aa8c940212ed1e61f45c92832ccfddba0b54ed8b72e2b33b8cc4004 WatchSource:0}: Error finding container 673fef636aa8c940212ed1e61f45c92832ccfddba0b54ed8b72e2b33b8cc4004: Status 404 returned error can't find the container with id 673fef636aa8c940212ed1e61f45c92832ccfddba0b54ed8b72e2b33b8cc4004
Jul 28 19:37:30.081283 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.081209    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:30.086131 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.086083    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6"]
Jul 28 19:37:30.095311 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:30.095285    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1dc6dad_e3e3_4200_898a_3dfa57fc10ea.slice/crio-3826b70f31a861ed2c3f1951a3d3b4fbe48e317320ec121e478f70253bb17443 WatchSource:0}: Error finding container 3826b70f31a861ed2c3f1951a3d3b4fbe48e317320ec121e478f70253bb17443: Status 404 returned error can't find the container with id 3826b70f31a861ed2c3f1951a3d3b4fbe48e317320ec121e478f70253bb17443
Jul 28 19:37:30.140406 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.140380    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj"]
Jul 28 19:37:30.174858 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.174807    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:30.268449 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.268342    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:30.328415 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.328388    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:30.355399 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.355334    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6" event={"ID":"f1dc6dad-e3e3-4200-898a-3dfa57fc10ea","Type":"ContainerStarted","Data":"3826b70f31a861ed2c3f1951a3d3b4fbe48e317320ec121e478f70253bb17443"}
Jul 28 19:37:30.359104 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.358512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkh9g" event={"ID":"62542e05-f80b-4490-9213-693a171a19d6","Type":"ContainerStarted","Data":"53c71e00acbb280d790470dbf44bfd6a26500e32c806251596961de269d93c7f"}
Jul 28 19:37:30.359104 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.358561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkh9g" event={"ID":"62542e05-f80b-4490-9213-693a171a19d6","Type":"ContainerStarted","Data":"45eec5106d354915629dd55739f42e880fbfd0d1c2047600e0d458f4f0d1d292"}
Jul 28 19:37:30.362936 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.362172    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5" event={"ID":"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37","Type":"ContainerStarted","Data":"673fef636aa8c940212ed1e61f45c92832ccfddba0b54ed8b72e2b33b8cc4004"}
Jul 28 19:37:30.598813 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:30.598723    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:30.871107 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:30.871028    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86f12d38_15f5_4721_a27a_5f2935352c70.slice/crio-84df05239f0b227ac18fb54afe7e9186bec6e20e6e4c529c51f0c560d182ed3b WatchSource:0}: Error finding container 84df05239f0b227ac18fb54afe7e9186bec6e20e6e4c529c51f0c560d182ed3b: Status 404 returned error can't find the container with id 84df05239f0b227ac18fb54afe7e9186bec6e20e6e4c529c51f0c560d182ed3b
Jul 28 19:37:31.001794 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:31.001689    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:31.100389 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:31.100362    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:31.371876 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:31.371832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj" event={"ID":"86f12d38-15f5-4721-a27a-5f2935352c70","Type":"ContainerStarted","Data":"84df05239f0b227ac18fb54afe7e9186bec6e20e6e4c529c51f0c560d182ed3b"}
Jul 28 19:37:31.532846 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:31.532765    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:31.674785 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:31.674710    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:31.795965 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:31.795930    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:31.898339 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:31.898313    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:31.998333 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:31.998246    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:32.205820 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:32.205781    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:32.305908 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:32.305871    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:32.833740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:32.833708    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:33.457198 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:33.457142    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:33.460297 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:33.460271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ea2f8539-0a89-4894-8bd6-73822815662e-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnll5\" (UID: \"ea2f8539-0a89-4894-8bd6-73822815662e\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:33.704578 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:33.704545    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ltwr9\""
Jul 28 19:37:33.711658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:33.711585    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"
Jul 28 19:37:35.266258 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:35.266214    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:35.330533 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:35.330501    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:35.385599 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:35.385576    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:37.251624 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:37.251588    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:37.353062 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:37.353033    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:37.432179 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:37.432144    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:37.739353 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:37.739325    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:39.182917 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.182771    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dkh9g" podStartSLOduration=75.647760816 podStartE2EDuration="1m17.182750994s" totalImagesPullingTime="1.534990178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:30.378910144 +0000 UTC m=+68.839668745" watchObservedRunningTime="2026-07-28 19:37:39.182750994 +0000 UTC m=+77.643509593"
Jul 28 19:37:39.184162 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.184086    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qnll5"]
Jul 28 19:37:39.401927 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.401863    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5" event={"ID":"cabaeed5-a3dd-440e-a8b0-df4ea3cc9c37","Type":"ContainerStarted","Data":"f6884fc097dd214d0409c457dde967ca5ade3c86fd0121fb9a683c291e0a530b"}
Jul 28 19:37:39.402082 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.402039    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:39.403812 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.403791    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5"
Jul 28 19:37:39.403999 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.403977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gq6td" event={"ID":"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5","Type":"ContainerStarted","Data":"b152da0945c221e876dd06be0e0ec6169403862c89df8d8db2541c1277a788b8"}
Jul 28 19:37:39.405663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.405635    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlzp2" event={"ID":"ac2f6cd6-d537-45e7-b8a5-03034fd92b25","Type":"ContainerStarted","Data":"0ed6b3beccddce99d4f52b3c0efd50b240a0bdd3683e671320796ff80e5c4064"}
Jul 28 19:37:39.405771 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.405689    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:37:39.405771 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.405715    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tlzp2" event={"ID":"ac2f6cd6-d537-45e7-b8a5-03034fd92b25","Type":"ContainerStarted","Data":"547b87003bca5559af248971639deef9e814a85e682dbdef48e57c6623b2b7b5"}
Jul 28 19:37:39.407548 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.407526    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj" event={"ID":"86f12d38-15f5-4721-a27a-5f2935352c70","Type":"ContainerStarted","Data":"3816a7daa4bb878a93c15aac8a9faba87e385a330f9483c8ec0c55e73c9d4623"}
Jul 28 19:37:39.407675 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.407554    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj" event={"ID":"86f12d38-15f5-4721-a27a-5f2935352c70","Type":"ContainerStarted","Data":"31829776d320e166ac7911bbd5ef7d3dbb49d360bac2e7c37d23fb77b2f4fb7e"}
Jul 28 19:37:39.407675 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.407569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj" event={"ID":"86f12d38-15f5-4721-a27a-5f2935352c70","Type":"ContainerStarted","Data":"e3dae24d7020cbfedaf87996b847c90cc82f76320c46ee809b7bad187060ae84"}
Jul 28 19:37:39.408572 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.408553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5" event={"ID":"ea2f8539-0a89-4894-8bd6-73822815662e","Type":"ContainerStarted","Data":"aaee2851c3017e9c4b86adf665dc3c1169c391d4a33c72cc1dad2b0f90cf9406"}
Jul 28 19:37:39.409875 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.409853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9nkx9" event={"ID":"464229a3-6aff-4f24-ba21-64b65a2182a4","Type":"ContainerStarted","Data":"c8607dafc0211cbc0f4b4a6e7a03738341573b6d7353c82166ed4a089573b5b6"}
Jul 28 19:37:39.411256 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.411233    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jfjzl" event={"ID":"6af25eb8-7d06-493e-8d26-343b128261e1","Type":"ContainerStarted","Data":"fca916c32782b04e648725a63b8d9ed14ba4be812a2209cc96f6052e4d504f5a"}
Jul 28 19:37:39.411369 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.411355    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:37:39.418740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.418706    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5fd95867b4-ng2j5" podStartSLOduration=1.437418803 podStartE2EDuration="10.418694808s" totalImagesPullingTime="8.981276005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:39.417597655 +0000 UTC m=+77.878356255" watchObservedRunningTime="2026-07-28 19:37:39.418694808 +0000 UTC m=+77.879453407"
Jul 28 19:37:39.469917 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.469863    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8487f78744-9dbhj" podStartSLOduration=2.296454026 podStartE2EDuration="10.469848188s" totalImagesPullingTime="8.173394162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:39.469271438 +0000 UTC m=+77.930030037" watchObservedRunningTime="2026-07-28 19:37:39.469848188 +0000 UTC m=+77.930606786"
Jul 28 19:37:39.485715 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.485675    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9nkx9" podStartSLOduration=66.446777107 podStartE2EDuration="1m17.485663139s" totalImagesPullingTime="11.038886032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:39.48512705 +0000 UTC m=+77.945885650" watchObservedRunningTime="2026-07-28 19:37:39.485663139 +0000 UTC m=+77.946421738"
Jul 28 19:37:39.503615 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.503563    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tlzp2" podStartSLOduration=34.950186261 podStartE2EDuration="44.50353786s" totalImagesPullingTime="9.553351599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:39.501701678 +0000 UTC m=+77.962460275" watchObservedRunningTime="2026-07-28 19:37:39.50353786 +0000 UTC m=+77.964296451"
Jul 28 19:37:39.518388 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:39.518355    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jfjzl" podStartSLOduration=66.513361617 podStartE2EDuration="1m17.51834363s" totalImagesPullingTime="11.004982013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:39.517724993 +0000 UTC m=+77.978483606" watchObservedRunningTime="2026-07-28 19:37:39.51834363 +0000 UTC m=+77.979102246"
Jul 28 19:37:40.293196 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:40.293160    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:40.394232 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:40.394177    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:41.421122 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:41.421079    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5" event={"ID":"ea2f8539-0a89-4894-8bd6-73822815662e","Type":"ContainerStarted","Data":"cd123f4f24f15fdea381cfe9be1e27f65a50a78a9219af3821c529a50f06b754"}
Jul 28 19:37:41.439559 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:41.439511    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnll5" podStartSLOduration=39.243754559 podStartE2EDuration="40.439496279s" totalImagesPullingTime="1.19574172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:41.439105868 +0000 UTC m=+79.899864470" watchObservedRunningTime="2026-07-28 19:37:41.439496279 +0000 UTC m=+79.900254877"
Jul 28 19:37:41.567027 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:41.566991    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:41.667069 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:41.667042    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:43.435410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:43.435375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gq6td" event={"ID":"2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5","Type":"ContainerStarted","Data":"4bd9cf1aa42e00329e3d661b79df5fa6dd564c5bc534a5460298ee31a8030268"}
Jul 28 19:37:43.457273 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:43.457221    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gq6td" podStartSLOduration=17.894552436 podStartE2EDuration="37.457206331s" totalImagesPullingTime="19.562653895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:37:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:43.455524596 +0000 UTC m=+81.916283195" watchObservedRunningTime="2026-07-28 19:37:43.457206331 +0000 UTC m=+81.917964930"
Jul 28 19:37:44.092909 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:44.092858    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:44.148226 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:44.148193    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:44.454975 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:44.454909    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:44.559282 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:44.559260    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:45.123036 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:45.123009    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:45.267181 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:45.267154    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:45.328092 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:45.328062    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:45.384804 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:45.384737    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:45.493549 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:45.493523    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:45.596316 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:45.596259    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:48.108935 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:48.108884    2555 patch_prober.go:28] interesting pod/image-registry-7889b89796-kjvfn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 19:37:48.109325 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:48.108962    2555 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7889b89796-kjvfn" podUID="42455482-d5c7-4560-967e-e4ee3568e209" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 19:37:48.363772 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:48.363742    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:48.452781 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:48.452747    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6" event={"ID":"f1dc6dad-e3e3-4200-898a-3dfa57fc10ea","Type":"ContainerStarted","Data":"0ba74a128760a9bdf4a82f6fb9738d89d3eddd35e5afc820cb06d40435bbcdd6"}
Jul 28 19:37:48.464848 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:48.464815    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:48.469323 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:48.469239    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5d58d5cfdb-d8fh6" podStartSLOduration=1.237076455 podStartE2EDuration="19.469222909s" totalImagesPullingTime="18.232146454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:48.468631172 +0000 UTC m=+86.929389765" watchObservedRunningTime="2026-07-28 19:37:48.469222909 +0000 UTC m=+86.929981511"
Jul 28 19:37:48.564968 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:48.564907    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:49.353989 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:49.353958    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:37:49.417777 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:49.417743    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tlzp2"
Jul 28 19:37:49.617557 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:49.617469    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:49.718352 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:49.718325    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:49.785046 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:49.785020    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:50.264706 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.264679    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:50.366207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.366176    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:50.613292 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.613264    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:50.860481 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.860454    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:50.939031 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.938964    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2fhj7"]
Jul 28 19:37:50.942250 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.942231    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:50.945536 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.945503    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 19:37:50.945650 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.945555    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 19:37:50.945650 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.945556    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pwz2m\""
Jul 28 19:37:50.946585 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.946569    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 19:37:50.946681 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.946602    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 19:37:50.947057 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.947040    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 19:37:50.947130 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.947078    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 19:37:50.971066 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:50.971044    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:51.060461 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.060442    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:51.081564 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081541    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42vwv\" (UniqueName: \"kubernetes.io/projected/b72cac4a-87d7-4021-a509-ec0dbccacedc-kube-api-access-42vwv\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.081676 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081572    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b72cac4a-87d7-4021-a509-ec0dbccacedc-metrics-client-ca\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.081676 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081591    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-root\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.081676 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-sys\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.081676 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081663    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.081871 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-wtmp\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.081871 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-tls\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.081871 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081754    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.081871 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.081776    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-textfile\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.165576 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.165557    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:51.182882 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.182860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-textfile\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183001 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.182905    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42vwv\" (UniqueName: \"kubernetes.io/projected/b72cac4a-87d7-4021-a509-ec0dbccacedc-kube-api-access-42vwv\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183001 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.182940    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b72cac4a-87d7-4021-a509-ec0dbccacedc-metrics-client-ca\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183001 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.182964    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-root\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183001 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.182992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-sys\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183070    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-wtmp\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-tls\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183130    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183207 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-root\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183458 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183269    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-textfile\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183458 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183290    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-wtmp\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183458 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183363    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b72cac4a-87d7-4021-a509-ec0dbccacedc-sys\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183610 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183557    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b72cac4a-87d7-4021-a509-ec0dbccacedc-metrics-client-ca\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.183701 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.183667    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.186083 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.186054    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.186252 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.186230    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b72cac4a-87d7-4021-a509-ec0dbccacedc-node-exporter-tls\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.190804 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.190750    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42vwv\" (UniqueName: \"kubernetes.io/projected/b72cac4a-87d7-4021-a509-ec0dbccacedc-kube-api-access-42vwv\") pod \"node-exporter-2fhj7\" (UID: \"b72cac4a-87d7-4021-a509-ec0dbccacedc\") " pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.251720 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.251699    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2fhj7"
Jul 28 19:37:51.262327 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:37:51.262306    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72cac4a_87d7_4021_a509_ec0dbccacedc.slice/crio-5c2d9f211ba0e3299ef7c98a23b10102a3349cacc8353a3c8999d3052bf7c955 WatchSource:0}: Error finding container 5c2d9f211ba0e3299ef7c98a23b10102a3349cacc8353a3c8999d3052bf7c955: Status 404 returned error can't find the container with id 5c2d9f211ba0e3299ef7c98a23b10102a3349cacc8353a3c8999d3052bf7c955
Jul 28 19:37:51.263620 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.263524    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:51.469060 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.468998    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2fhj7" event={"ID":"b72cac4a-87d7-4021-a509-ec0dbccacedc","Type":"ContainerStarted","Data":"5c2d9f211ba0e3299ef7c98a23b10102a3349cacc8353a3c8999d3052bf7c955"}
Jul 28 19:37:51.841596 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.841557    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:51.942669 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:51.942604    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:52.243166 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:52.243089    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:52.474050 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:52.474014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2fhj7" event={"ID":"b72cac4a-87d7-4021-a509-ec0dbccacedc","Type":"ContainerStarted","Data":"dd01ccfcdfe30fcb5986edff43b2086a0fd304f1a3bb8cd97a33e56341e0da26"}
Jul 28 19:37:53.296259 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:53.296226    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:53.402055 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:53.402029    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:53.480118 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:53.480086    2555 generic.go:356] "Generic (PLEG): container finished" podID="b72cac4a-87d7-4021-a509-ec0dbccacedc" containerID="dd01ccfcdfe30fcb5986edff43b2086a0fd304f1a3bb8cd97a33e56341e0da26" exitCode=0
Jul 28 19:37:53.480472 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:53.480195    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2fhj7" event={"ID":"b72cac4a-87d7-4021-a509-ec0dbccacedc","Type":"ContainerDied","Data":"dd01ccfcdfe30fcb5986edff43b2086a0fd304f1a3bb8cd97a33e56341e0da26"}
Jul 28 19:37:53.505554 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:53.505529    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:53.790253 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:53.790219    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:53.881359 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:53.881289    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:54.001602 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:54.001573    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:54.328004 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:54.327970    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:54.485739 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:54.485706    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2fhj7" event={"ID":"b72cac4a-87d7-4021-a509-ec0dbccacedc","Type":"ContainerStarted","Data":"ffb7d840e2ddecf338c74a3d756dd5b158f6d67c05240a311a9a89758ad2868a"}
Jul 28 19:37:54.486183 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:54.485769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2fhj7" event={"ID":"b72cac4a-87d7-4021-a509-ec0dbccacedc","Type":"ContainerStarted","Data":"974b4e2417450e3095916bf192455f5b4f797e877f74713ae1cdfbf07e653ff8"}
Jul 28 19:37:54.506984 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:54.506936    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2fhj7" podStartSLOduration=3.361434566 podStartE2EDuration="4.506922593s" totalImagesPullingTime="1.084422015s" totalInitContainerRuntime="61.066012ms" isStatefulPod=true podCreationTimestamp="2026-07-28 19:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:37:54.505875502 +0000 UTC m=+92.966634103" watchObservedRunningTime="2026-07-28 19:37:54.506922593 +0000 UTC m=+92.967681193"
Jul 28 19:37:55.614124 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:55.614091    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:55.687940 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:55.687877    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:55.850832 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:55.850808    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:56.842499 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:56.842466    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:56.935911 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:56.935849    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:57.197296 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:57.197220    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:59.118161 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:59.118126    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:59.225377 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:59.225350    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:59.714707 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:59.714677    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7889b89796-kjvfn"]
Jul 28 19:37:59.791288 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:59.791262    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:37:59.889253 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:37:59.889228    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:00.314505 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:00.314477    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:00.410266 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:00.410240    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:01.507163 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:01.507127    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:01.606797 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:01.606766    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:01.906328 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:01.906300    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:02.007347 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:02.007321    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:03.298877 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:03.298849    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:03.399182 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:03.399156    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:05.259517 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:05.259472    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:05.363548 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:05.363512    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:05.806295 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:05.806253    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:05.906677 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:05.906649    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:07.598463 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:07.598431    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:07.665361 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:07.665339    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:07.722865 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:07.722839    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:09.445665 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:09.445620    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:09.510227 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:09.510201    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:10.418572 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:10.418545    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jfjzl"
Jul 28 19:38:20.260518 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:20.260477    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:20.370080 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:20.370055    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:21.551244 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:21.551203    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:24.733556 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:24.733507    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7889b89796-kjvfn" podUID="42455482-d5c7-4560-967e-e4ee3568e209" containerName="registry" containerID="cri-o://3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563" gracePeriod=30
Jul 28 19:38:24.975355 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:24.975333    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:38:25.030789 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.030737    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:25.079760 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.079738    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:38:25.106000 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.105984    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5vr7\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-kube-api-access-s5vr7\") pod \"42455482-d5c7-4560-967e-e4ee3568e209\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") "
Jul 28 19:38:25.106053 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106017    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") pod \"42455482-d5c7-4560-967e-e4ee3568e209\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") "
Jul 28 19:38:25.106053 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106039    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-trusted-ca\") pod \"42455482-d5c7-4560-967e-e4ee3568e209\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") "
Jul 28 19:38:25.106127 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106070    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42455482-d5c7-4560-967e-e4ee3568e209-ca-trust-extracted\") pod \"42455482-d5c7-4560-967e-e4ee3568e209\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") "
Jul 28 19:38:25.106173 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106125    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-registry-certificates\") pod \"42455482-d5c7-4560-967e-e4ee3568e209\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") "
Jul 28 19:38:25.106173 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106159    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-image-registry-private-configuration\") pod \"42455482-d5c7-4560-967e-e4ee3568e209\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") "
Jul 28 19:38:25.106273 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106193    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-installation-pull-secrets\") pod \"42455482-d5c7-4560-967e-e4ee3568e209\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") "
Jul 28 19:38:25.106273 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106252    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-bound-sa-token\") pod \"42455482-d5c7-4560-967e-e4ee3568e209\" (UID: \"42455482-d5c7-4560-967e-e4ee3568e209\") "
Jul 28 19:38:25.106532 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106484    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "42455482-d5c7-4560-967e-e4ee3568e209" (UID: "42455482-d5c7-4560-967e-e4ee3568e209"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:38:25.106532 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.106518    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "42455482-d5c7-4560-967e-e4ee3568e209" (UID: "42455482-d5c7-4560-967e-e4ee3568e209"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:38:25.108628 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.108589    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "42455482-d5c7-4560-967e-e4ee3568e209" (UID: "42455482-d5c7-4560-967e-e4ee3568e209"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:38:25.108727 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.108681    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "42455482-d5c7-4560-967e-e4ee3568e209" (UID: "42455482-d5c7-4560-967e-e4ee3568e209"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:38:25.108727 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.108696    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "42455482-d5c7-4560-967e-e4ee3568e209" (UID: "42455482-d5c7-4560-967e-e4ee3568e209"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:38:25.108821 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.108789    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "42455482-d5c7-4560-967e-e4ee3568e209" (UID: "42455482-d5c7-4560-967e-e4ee3568e209"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:38:25.108936 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.108914    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-kube-api-access-s5vr7" (OuterVolumeSpecName: "kube-api-access-s5vr7") pod "42455482-d5c7-4560-967e-e4ee3568e209" (UID: "42455482-d5c7-4560-967e-e4ee3568e209"). InnerVolumeSpecName "kube-api-access-s5vr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:38:25.114234 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.114213    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42455482-d5c7-4560-967e-e4ee3568e209-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "42455482-d5c7-4560-967e-e4ee3568e209" (UID: "42455482-d5c7-4560-967e-e4ee3568e209"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:38:25.206693 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.206671    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-bound-sa-token\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:38:25.206693 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.206691    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5vr7\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-kube-api-access-s5vr7\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:38:25.206805 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.206701    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42455482-d5c7-4560-967e-e4ee3568e209-registry-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:38:25.206805 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.206710    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-trusted-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:38:25.206805 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.206718    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42455482-d5c7-4560-967e-e4ee3568e209-ca-trust-extracted\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:38:25.206805 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.206725    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42455482-d5c7-4560-967e-e4ee3568e209-registry-certificates\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:38:25.206805 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.206735    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-image-registry-private-configuration\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:38:25.206805 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.206745    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42455482-d5c7-4560-967e-e4ee3568e209-installation-pull-secrets\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:38:25.594858 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.594823    2555 generic.go:356] "Generic (PLEG): container finished" podID="42455482-d5c7-4560-967e-e4ee3568e209" containerID="3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563" exitCode=0
Jul 28 19:38:25.594991 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.594872    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7889b89796-kjvfn"
Jul 28 19:38:25.594991 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.594927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7889b89796-kjvfn" event={"ID":"42455482-d5c7-4560-967e-e4ee3568e209","Type":"ContainerDied","Data":"3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563"}
Jul 28 19:38:25.595065 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.594994    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7889b89796-kjvfn" event={"ID":"42455482-d5c7-4560-967e-e4ee3568e209","Type":"ContainerDied","Data":"cf6f0d55571754adc0f1eb27d26ee3dbbbefdf8ea2da59cc8934fa204820305b"}
Jul 28 19:38:25.595065 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.595018    2555 scope.go:117] "RemoveContainer" containerID="3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563"
Jul 28 19:38:25.605973 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.605925    2555 scope.go:117] "RemoveContainer" containerID="3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563"
Jul 28 19:38:25.606295 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:38:25.606268    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563\": container with ID starting with 3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563 not found: ID does not exist" containerID="3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563"
Jul 28 19:38:25.606368 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.606302    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563"} err="failed to get container status \"3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563\": rpc error: code = NotFound desc = could not find container \"3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563\": container with ID starting with 3613ae393640b94aebaeb1d51d28984d3645ff34471b6618a6c84de33841b563 not found: ID does not exist"
Jul 28 19:38:25.623239 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.623216    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7889b89796-kjvfn"]
Jul 28 19:38:25.627560 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.627540    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7889b89796-kjvfn"]
Jul 28 19:38:25.983675 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:38:25.983609    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42455482-d5c7-4560-967e-e4ee3568e209" path="/var/lib/kubelet/pods/42455482-d5c7-4560-967e-e4ee3568e209/volumes"
Jul 28 19:39:10.125588 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:39:10.125555    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:39:10.242615 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:39:10.242588    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:39:19.901280 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:39:19.901189    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:39:31.002850 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:39:31.002814    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:39:31.103492 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:39:31.103459    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:39:31.204802 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:39:31.204771    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:39:47.414994 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:39:47.414936    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:39:47.480352 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:39:47.480302    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:25.380600 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:25.380554    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:25.444024 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:25.443987    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:29.868723 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:29.868679    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:29.973944 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:29.973914    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:37.330622 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:37.330586    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:37.401351 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:37.401310    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:41.312955 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:41.312924    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:41.378381 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:41.378353    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:41.436353 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:41.436331    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:50.269428 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:50.269340    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:40:50.371787 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:40:50.371757    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:17.184825 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:17.184787    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:17.317239 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:17.317191    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:21.908383 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:21.908343    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 19:41:44.043814 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:44.043781    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:45.318711 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:45.318672    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:47.617777 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:47.617732    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:47.674529 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:47.674494    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:49.040431 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:49.040400    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:49.173911 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:49.173858    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:49.454938 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:49.454911    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:49.572386 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:49.572359    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:51.643225 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:51.643193    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:52.325992 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:52.325961    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:54.418646 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:54.418588    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:54.488647 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:54.488612    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:59.862356 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:59.862305    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:41:59.960284 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:41:59.960249    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:00.271104 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:00.271038    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:00.432345 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:00.432314    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:00.525321 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:00.525258    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:07.228082 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:07.228043    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:07.347441 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:07.347413    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:09.634623 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:09.634588    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:09.747127 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:09.747094    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:21.270047 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:21.269954    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:21.386612 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:21.386576    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:21.681592 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:21.681566    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:21.931834 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:21.931776    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:22.037274 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.037248    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:22.076840 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.076815    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"]
Jul 28 19:42:22.077053 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.077041    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42455482-d5c7-4560-967e-e4ee3568e209" containerName="registry"
Jul 28 19:42:22.077105 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.077054    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42455482-d5c7-4560-967e-e4ee3568e209" containerName="registry"
Jul 28 19:42:22.077142 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.077105    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42455482-d5c7-4560-967e-e4ee3568e209" containerName="registry"
Jul 28 19:42:22.080001 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.079985    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.084340 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.084314    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-25r4d\""
Jul 28 19:42:22.084439 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.084381    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 19:42:22.084596 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.084580    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 19:42:22.084712 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.084656    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 19:42:22.084712 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.084670    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 19:42:22.114487 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.114460    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"]
Jul 28 19:42:22.132379 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.132355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-webhook-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.132480 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.132395    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-tmp\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.132480 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.132422    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.132564 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.132511    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-manifests\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.132564 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.132542    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn75w\" (UniqueName: \"kubernetes.io/projected/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-kube-api-access-gn75w\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.233836 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.233778    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-manifests\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.233836 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.233814    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn75w\" (UniqueName: \"kubernetes.io/projected/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-kube-api-access-gn75w\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.234010 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.233852    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-webhook-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.234010 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.233900    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-tmp\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.234010 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.233925    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.234231 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.234206    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-manifests\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.236164 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.236146    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-tmp\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.236374 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.236351    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-webhook-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.236604 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.236588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.243452 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.243432    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn75w\" (UniqueName: \"kubernetes.io/projected/fabb0aa1-2cac-421b-a8b4-8a86b39ea62f-kube-api-access-gn75w\") pod \"opendatahub-operator-controller-manager-856746f77b-q8wg9\" (UID: \"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f\") " pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.278372 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.278341    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:22.379810 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.379783    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:22.388736 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.388709    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:22.485694 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.485549    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:22.593223 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.593197    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"]
Jul 28 19:42:22.596514 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:42:22.596491    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabb0aa1_2cac_421b_a8b4_8a86b39ea62f.slice/crio-c69867938a1d736a38dabc11800e1eb323382a46d8c841b4d5e1855bfdce5398 WatchSource:0}: Error finding container c69867938a1d736a38dabc11800e1eb323382a46d8c841b4d5e1855bfdce5398: Status 404 returned error can't find the container with id c69867938a1d736a38dabc11800e1eb323382a46d8c841b4d5e1855bfdce5398
Jul 28 19:42:22.598118 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.598101    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 19:42:22.669879 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.669858    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:22.773368 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:22.773303    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:23.424121 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:23.424076    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9" event={"ID":"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f","Type":"ContainerStarted","Data":"c69867938a1d736a38dabc11800e1eb323382a46d8c841b4d5e1855bfdce5398"}
Jul 28 19:42:26.436814 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:26.436775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9" event={"ID":"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f","Type":"ContainerStarted","Data":"3909eb5e20baba90567fc6739ce8a71e90fe6b46751cc1fa3602fad562c27597"}
Jul 28 19:42:27.442445 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:27.442407    2555 generic.go:356] "Generic (PLEG): container finished" podID="fabb0aa1-2cac-421b-a8b4-8a86b39ea62f" containerID="3909eb5e20baba90567fc6739ce8a71e90fe6b46751cc1fa3602fad562c27597" exitCode=0
Jul 28 19:42:27.442844 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:27.442457    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9" event={"ID":"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f","Type":"ContainerDied","Data":"3909eb5e20baba90567fc6739ce8a71e90fe6b46751cc1fa3602fad562c27597"}
Jul 28 19:42:28.447897 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:28.447846    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9" event={"ID":"fabb0aa1-2cac-421b-a8b4-8a86b39ea62f","Type":"ContainerStarted","Data":"246ab573f17e3e57e5a6de935d1a6d7c70124af101082a82b68ac918a7cffce1"}
Jul 28 19:42:28.448301 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:28.447999    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:28.473144 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:28.473093    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9" podStartSLOduration=2.173919804 podStartE2EDuration="6.473074706s" totalImagesPullingTime="3.279633208s" totalInitContainerRuntime="1.019521694s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:28.47092368 +0000 UTC m=+366.931682280" watchObservedRunningTime="2026-07-28 19:42:28.473074706 +0000 UTC m=+366.933833305"
Jul 28 19:42:33.842644 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:33.842613    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:33.946398 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:33.946366    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:39.454373 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:39.454342    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-856746f77b-q8wg9"
Jul 28 19:42:41.614554 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:41.614519    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:41.777194 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:41.777152    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:42.316510 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.316480    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:42.472340 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.472311    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"]
Jul 28 19:42:42.481681 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.481660    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.485586 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.485569    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 19:42:42.488451 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.488437    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 19:42:42.491898 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.491872    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 19:42:42.496169 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.496153    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 19:42:42.500654 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.500633    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"]
Jul 28 19:42:42.528500 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.528478    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:42.570523 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.570472    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2ls\" (UniqueName: \"kubernetes.io/projected/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-kube-api-access-wp2ls\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.570603 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.570524    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-tls-certs\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.570603 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.570540    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-tmp\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.671823 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.671798    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-tls-certs\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.672173 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.671828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-tmp\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.672173 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.671860    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2ls\" (UniqueName: \"kubernetes.io/projected/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-kube-api-access-wp2ls\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.674144 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.674124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-tmp\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.674282 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.674261    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-tls-certs\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.716670 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.716639    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2ls\" (UniqueName: \"kubernetes.io/projected/fa904a59-c4b1-46aa-a3f6-f4df75ebc83c-kube-api-access-wp2ls\") pod \"kube-auth-proxy-77597fcbdb-sr9kb\" (UID: \"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c\") " pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.790502 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.790479    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"
Jul 28 19:42:42.807713 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.807690    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6s5zb"]
Jul 28 19:42:42.809664 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.809648    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:42.815542 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:42:42.815514    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"odh-model-controller-7998c67dc4-6s5zb\" is forbidden: User \"system:node:ip-10-0-138-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-173.ec2.internal' and this object" podUID="a9cb6c87-c689-4ed6-9ce2-4371cd243859" pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:42.818208 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:42:42.818177    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-dockercfg-cm89c\" is forbidden: User \"system:node:ip-10-0-138-173.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-173.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cm89c\"" type="*v1.Secret"
Jul 28 19:42:42.819217 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:42:42.819186    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-138-173.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-173.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret"
Jul 28 19:42:42.845194 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.844373    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6s5zb"]
Jul 28 19:42:42.873951 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.873920    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cb6c87-c689-4ed6-9ce2-4371cd243859-cert\") pod \"odh-model-controller-7998c67dc4-6s5zb\" (UID: \"a9cb6c87-c689-4ed6-9ce2-4371cd243859\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:42.874094 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.873983    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jdk\" (UniqueName: \"kubernetes.io/projected/a9cb6c87-c689-4ed6-9ce2-4371cd243859-kube-api-access-79jdk\") pod \"odh-model-controller-7998c67dc4-6s5zb\" (UID: \"a9cb6c87-c689-4ed6-9ce2-4371cd243859\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:42.910309 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.910274    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"]
Jul 28 19:42:42.912245 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.912229    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:42.916087 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.916068    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 19:42:42.919604 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.919581    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 19:42:42.919681 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.919584    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:42:42.919866 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.919854    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5sxt8\""
Jul 28 19:42:42.920315 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.920300    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 19:42:42.920646 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.920634    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 19:42:42.933162 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.933136    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"]
Jul 28 19:42:42.952303 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.952287    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:42.974748 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.974724    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c69932b-85ed-4c83-9290-5e3205ccd6fc-cert\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:42.974836 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.974765    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jdk\" (UniqueName: \"kubernetes.io/projected/a9cb6c87-c689-4ed6-9ce2-4371cd243859-kube-api-access-79jdk\") pod \"odh-model-controller-7998c67dc4-6s5zb\" (UID: \"a9cb6c87-c689-4ed6-9ce2-4371cd243859\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:42.974836 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.974800    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5szh\" (UniqueName: \"kubernetes.io/projected/5c69932b-85ed-4c83-9290-5e3205ccd6fc-kube-api-access-g5szh\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:42.974926 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.974880    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c69932b-85ed-4c83-9290-5e3205ccd6fc-metrics-cert\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:42.974966 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.974941    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c69932b-85ed-4c83-9290-5e3205ccd6fc-manager-config\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:42.975001 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.974962    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cb6c87-c689-4ed6-9ce2-4371cd243859-cert\") pod \"odh-model-controller-7998c67dc4-6s5zb\" (UID: \"a9cb6c87-c689-4ed6-9ce2-4371cd243859\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:42.983501 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:42:42.983479    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa904a59_c4b1_46aa_a3f6_f4df75ebc83c.slice/crio-d19e55f9a8ff9233f55ccd5b5bf8a595041fd55abe13880cdc5d0646e992015b WatchSource:0}: Error finding container d19e55f9a8ff9233f55ccd5b5bf8a595041fd55abe13880cdc5d0646e992015b: Status 404 returned error can't find the container with id d19e55f9a8ff9233f55ccd5b5bf8a595041fd55abe13880cdc5d0646e992015b
Jul 28 19:42:42.991536 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:42.991515    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb"]
Jul 28 19:42:43.004595 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.004577    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jdk\" (UniqueName: \"kubernetes.io/projected/a9cb6c87-c689-4ed6-9ce2-4371cd243859-kube-api-access-79jdk\") pod \"odh-model-controller-7998c67dc4-6s5zb\" (UID: \"a9cb6c87-c689-4ed6-9ce2-4371cd243859\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:43.076142 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.076112    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5szh\" (UniqueName: \"kubernetes.io/projected/5c69932b-85ed-4c83-9290-5e3205ccd6fc-kube-api-access-g5szh\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.076249 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.076162    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c69932b-85ed-4c83-9290-5e3205ccd6fc-metrics-cert\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.076249 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.076194    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c69932b-85ed-4c83-9290-5e3205ccd6fc-manager-config\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.076360 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.076249    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c69932b-85ed-4c83-9290-5e3205ccd6fc-cert\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.076954 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.076931    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5c69932b-85ed-4c83-9290-5e3205ccd6fc-manager-config\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.078471 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.078430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c69932b-85ed-4c83-9290-5e3205ccd6fc-cert\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.078544 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.078475    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5c69932b-85ed-4c83-9290-5e3205ccd6fc-metrics-cert\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.101603 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.101550    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5szh\" (UniqueName: \"kubernetes.io/projected/5c69932b-85ed-4c83-9290-5e3205ccd6fc-kube-api-access-g5szh\") pod \"lws-controller-manager-cbcf9f5f9-gwcb6\" (UID: \"5c69932b-85ed-4c83-9290-5e3205ccd6fc\") " pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.120224 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.120199    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:43.221138 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.221114    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:43.253290 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.253268    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:43.349495 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.349468    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"]
Jul 28 19:42:43.352101 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:42:43.352052    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c69932b_85ed_4c83_9290_5e3205ccd6fc.slice/crio-ffff3b81207430198b3b592114f3c884d957127268229a67c2e836b515649925 WatchSource:0}: Error finding container ffff3b81207430198b3b592114f3c884d957127268229a67c2e836b515649925: Status 404 returned error can't find the container with id ffff3b81207430198b3b592114f3c884d957127268229a67c2e836b515649925
Jul 28 19:42:43.503240 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.503209    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6" event={"ID":"5c69932b-85ed-4c83-9290-5e3205ccd6fc","Type":"ContainerStarted","Data":"ffff3b81207430198b3b592114f3c884d957127268229a67c2e836b515649925"}
Jul 28 19:42:43.504271 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.504250    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb" event={"ID":"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c","Type":"ContainerStarted","Data":"d19e55f9a8ff9233f55ccd5b5bf8a595041fd55abe13880cdc5d0646e992015b"}
Jul 28 19:42:43.739630 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.739557    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cm89c\""
Jul 28 19:42:43.793475 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.793444    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:43.975854 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:42:43.975814    2555 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: failed to sync secret cache: timed out waiting for the condition
Jul 28 19:42:43.976014 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:42:43.975954    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9cb6c87-c689-4ed6-9ce2-4371cd243859-cert podName:a9cb6c87-c689-4ed6-9ce2-4371cd243859 nodeName:}" failed. No retries permitted until 2026-07-28 19:42:44.475876273 +0000 UTC m=+382.936634855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a9cb6c87-c689-4ed6-9ce2-4371cd243859-cert") pod "odh-model-controller-7998c67dc4-6s5zb" (UID: "a9cb6c87-c689-4ed6-9ce2-4371cd243859") : failed to sync secret cache: timed out waiting for the condition
Jul 28 19:42:43.996164 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:43.996069    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:44.116339 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:44.116312    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 28 19:42:44.321097 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:44.321070    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:44.487538 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:44.487506    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cb6c87-c689-4ed6-9ce2-4371cd243859-cert\") pod \"odh-model-controller-7998c67dc4-6s5zb\" (UID: \"a9cb6c87-c689-4ed6-9ce2-4371cd243859\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:44.489861 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:44.489843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9cb6c87-c689-4ed6-9ce2-4371cd243859-cert\") pod \"odh-model-controller-7998c67dc4-6s5zb\" (UID: \"a9cb6c87-c689-4ed6-9ce2-4371cd243859\") " pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:44.579296 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:44.579226    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:44.621530 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:44.621500    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:44.662310 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:44.662283    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:45.289261 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:45.289238    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6s5zb"]
Jul 28 19:42:45.292230 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:42:45.292205    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cb6c87_c689_4ed6_9ce2_4371cd243859.slice/crio-5fd17baa5c96457bbf419a0e2970c2fa1cc78be9e144e470e32bfa3cf6f1aa18 WatchSource:0}: Error finding container 5fd17baa5c96457bbf419a0e2970c2fa1cc78be9e144e470e32bfa3cf6f1aa18: Status 404 returned error can't find the container with id 5fd17baa5c96457bbf419a0e2970c2fa1cc78be9e144e470e32bfa3cf6f1aa18
Jul 28 19:42:45.462433 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:45.462377    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:45.515811 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:45.515782    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb" event={"ID":"fa904a59-c4b1-46aa-a3f6-f4df75ebc83c","Type":"ContainerStarted","Data":"c2d484f48e5659a5452b63f1deca9e4501fd7e5b8d60e4379432a344eaac01a6"}
Jul 28 19:42:45.516918 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:45.516882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb" event={"ID":"a9cb6c87-c689-4ed6-9ce2-4371cd243859","Type":"ContainerStarted","Data":"5fd17baa5c96457bbf419a0e2970c2fa1cc78be9e144e470e32bfa3cf6f1aa18"}
Jul 28 19:42:45.534014 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:45.533964    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77597fcbdb-sr9kb" podStartSLOduration=1.299029325 podStartE2EDuration="3.533949437s" totalImagesPullingTime="2.234920112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:45.53350836 +0000 UTC m=+383.994266963" watchObservedRunningTime="2026-07-28 19:42:45.533949437 +0000 UTC m=+383.994708082"
Jul 28 19:42:45.539177 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:45.539140    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:45.925822 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:45.925680    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:48.529235 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:48.529205    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb" event={"ID":"a9cb6c87-c689-4ed6-9ce2-4371cd243859","Type":"ContainerStarted","Data":"2913162c1b79c88c3abfb86e9c35c4351a7b642aeec8d7dc613102fe400395c5"}
Jul 28 19:42:48.529630 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:48.529336    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:42:48.546330 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:48.546286    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb" podStartSLOduration=3.99886004 podStartE2EDuration="6.546271575s" totalImagesPullingTime="2.547411535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:48.545738035 +0000 UTC m=+387.006496646" watchObservedRunningTime="2026-07-28 19:42:48.546271575 +0000 UTC m=+387.007030175"
Jul 28 19:42:49.331731 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:49.331696    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:49.458128 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:49.458096    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:50.226840 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:50.226808    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:50.366229 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:50.366208    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:50.501934 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:50.501865    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:50.538124 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:50.538093    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6" event={"ID":"5c69932b-85ed-4c83-9290-5e3205ccd6fc","Type":"ContainerStarted","Data":"b89074eeaf01bb72ba1602924a00acc96cba2bac5320c64e9485ce95fb20d593"}
Jul 28 19:42:50.538214 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:50.538178    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:42:50.575729 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:50.575684    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6" podStartSLOduration=1.654711474 podStartE2EDuration="8.57566957s" totalImagesPullingTime="6.920958096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:42:50.574444714 +0000 UTC m=+389.035203312" watchObservedRunningTime="2026-07-28 19:42:50.57566957 +0000 UTC m=+389.036428170"
Jul 28 19:42:53.887925 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:53.887864    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:54.133599 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:54.133556    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:54.358303 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:54.358250    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:54.524002 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:54.523977    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:54.649868 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:54.649770    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:42:59.535504 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:42:59.535475    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-6s5zb"
Jul 28 19:43:01.543620 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:43:01.543582    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cbcf9f5f9-gwcb6"
Jul 28 19:43:29.566120 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:43:29.566090    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:43:43.078400 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:43:43.078354    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:43:43.177364 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:43:43.177337    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:43:43.273925 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:43:43.273898    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:43:43.997415 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:43:43.997341    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:43:44.206230 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:43:44.206200    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:02.808192 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:02.808142    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:02.908126 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:02.908084    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:15.884451 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:15.884395    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:20.104315 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:20.104275    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:22.281968 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:22.281940    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:22.379541 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:22.379503    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:22.517714 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:22.517687    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:22.689021 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:22.688997    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:23.943174 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:23.943119    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:27.090115 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:27.090079    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:27.501099 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:27.500995    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:27.724428 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:27.724385    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:30.543426 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:30.543395    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:31.900948 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:31.900908    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:31.999332 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:31.999305    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:33.351780 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:33.351723    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:33.457469 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:33.457437    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:33.549777 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:33.549750    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:34.897381 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:34.897345    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:35.101786 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:35.101755    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:35.776709 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:35.776659    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:35.899236 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:35.899198    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:39.156773 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:39.156732    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:39.258918 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:39.258849    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:47.531344 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:47.531286    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:47.636064 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:47.636032    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:49.983182 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:49.983135    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:50.086791 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:50.086762    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:52.882761 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:52.882723    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:53.024125 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:53.024098    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:56.373341 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:56.373291    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:44:56.475397 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:44:56.475364    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:05.213103 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:05.213069    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:05.353986 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:05.353950    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:07.810119 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:07.810075    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:07.918373 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:07.918341    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:11.029245 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:11.029208    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:11.129565 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:11.129532    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:14.616152 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:14.616040    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:14.673338 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:14.673307    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:19.381144 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:19.381110    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:24.093597 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:24.093558    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:24.209321 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:24.209294    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:25.088155 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:25.088105    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:25.479880 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:25.479788    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:29.590416 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:29.590372    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:29.642352 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:29.642318    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:39.297168 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.297138    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:39.413197 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.413161    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:39.477924 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.477873    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:39.783735 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.783660    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:45:39.787303 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.787282    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:39.789773 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.789743    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 19:45:39.789773 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.789757    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 19:45:39.791218 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.791194    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 19:45:39.791617 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.791228    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q24p6\""
Jul 28 19:45:39.797388 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.797350    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:45:39.808189 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.808164    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:45:39.876179 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.876154    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2c52c858-4373-4f1a-8cc3-4fc3f4037aac-config-file\") pod \"limitador-limitador-54d6d988d5-lwx8z\" (UID: \"2c52c858-4373-4f1a-8cc3-4fc3f4037aac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:39.876279 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.876194    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlfk\" (UniqueName: \"kubernetes.io/projected/2c52c858-4373-4f1a-8cc3-4fc3f4037aac-kube-api-access-9xlfk\") pod \"limitador-limitador-54d6d988d5-lwx8z\" (UID: \"2c52c858-4373-4f1a-8cc3-4fc3f4037aac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:39.976763 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.976738    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlfk\" (UniqueName: \"kubernetes.io/projected/2c52c858-4373-4f1a-8cc3-4fc3f4037aac-kube-api-access-9xlfk\") pod \"limitador-limitador-54d6d988d5-lwx8z\" (UID: \"2c52c858-4373-4f1a-8cc3-4fc3f4037aac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:39.976867 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.976792    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2c52c858-4373-4f1a-8cc3-4fc3f4037aac-config-file\") pod \"limitador-limitador-54d6d988d5-lwx8z\" (UID: \"2c52c858-4373-4f1a-8cc3-4fc3f4037aac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:39.977362 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.977342    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2c52c858-4373-4f1a-8cc3-4fc3f4037aac-config-file\") pod \"limitador-limitador-54d6d988d5-lwx8z\" (UID: \"2c52c858-4373-4f1a-8cc3-4fc3f4037aac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:39.988686 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:39.988663    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlfk\" (UniqueName: \"kubernetes.io/projected/2c52c858-4373-4f1a-8cc3-4fc3f4037aac-kube-api-access-9xlfk\") pod \"limitador-limitador-54d6d988d5-lwx8z\" (UID: \"2c52c858-4373-4f1a-8cc3-4fc3f4037aac\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:40.105395 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.105370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:40.113834 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.113812    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-tbtp8"]
Jul 28 19:45:40.117480 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.117454    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tbtp8"
Jul 28 19:45:40.120179 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.120130    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grx9k\""
Jul 28 19:45:40.125419 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.125395    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbtp8"]
Jul 28 19:45:40.144098 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.144077    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:40.202751 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.202687    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:40.236637 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.236613    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:45:40.239850 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:45:40.239825    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c52c858_4373_4f1a_8cc3_4fc3f4037aac.slice/crio-eaf3f7bae0a2646ce0b6811f1b8ac67a25bb9e51d6939e03dc06e3dcce36487f WatchSource:0}: Error finding container eaf3f7bae0a2646ce0b6811f1b8ac67a25bb9e51d6939e03dc06e3dcce36487f: Status 404 returned error can't find the container with id eaf3f7bae0a2646ce0b6811f1b8ac67a25bb9e51d6939e03dc06e3dcce36487f
Jul 28 19:45:40.278184 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.278159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxddt\" (UniqueName: \"kubernetes.io/projected/9911a8f7-3135-4081-b2ad-c9efbf7936b7-kube-api-access-sxddt\") pod \"authorino-58cd955675-tbtp8\" (UID: \"9911a8f7-3135-4081-b2ad-c9efbf7936b7\") " pod="kuadrant-system/authorino-58cd955675-tbtp8"
Jul 28 19:45:40.379199 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.379144    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxddt\" (UniqueName: \"kubernetes.io/projected/9911a8f7-3135-4081-b2ad-c9efbf7936b7-kube-api-access-sxddt\") pod \"authorino-58cd955675-tbtp8\" (UID: \"9911a8f7-3135-4081-b2ad-c9efbf7936b7\") " pod="kuadrant-system/authorino-58cd955675-tbtp8"
Jul 28 19:45:40.387009 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.386987    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxddt\" (UniqueName: \"kubernetes.io/projected/9911a8f7-3135-4081-b2ad-c9efbf7936b7-kube-api-access-sxddt\") pod \"authorino-58cd955675-tbtp8\" (UID: \"9911a8f7-3135-4081-b2ad-c9efbf7936b7\") " pod="kuadrant-system/authorino-58cd955675-tbtp8"
Jul 28 19:45:40.437156 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.437135    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tbtp8"
Jul 28 19:45:40.560679 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:40.560652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbtp8"]
Jul 28 19:45:40.563938 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:45:40.563915    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9911a8f7_3135_4081_b2ad_c9efbf7936b7.slice/crio-40a3e474f6d332c9e08c2e53bbccd0fd6d29cc8f2a591df5d7845d22d46c6ab2 WatchSource:0}: Error finding container 40a3e474f6d332c9e08c2e53bbccd0fd6d29cc8f2a591df5d7845d22d46c6ab2: Status 404 returned error can't find the container with id 40a3e474f6d332c9e08c2e53bbccd0fd6d29cc8f2a591df5d7845d22d46c6ab2
Jul 28 19:45:41.173945 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:41.173913    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tbtp8" event={"ID":"9911a8f7-3135-4081-b2ad-c9efbf7936b7","Type":"ContainerStarted","Data":"40a3e474f6d332c9e08c2e53bbccd0fd6d29cc8f2a591df5d7845d22d46c6ab2"}
Jul 28 19:45:41.175362 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:41.175331    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z" event={"ID":"2c52c858-4373-4f1a-8cc3-4fc3f4037aac","Type":"ContainerStarted","Data":"eaf3f7bae0a2646ce0b6811f1b8ac67a25bb9e51d6939e03dc06e3dcce36487f"}
Jul 28 19:45:44.995989 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:44.995953    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbtp8"]
Jul 28 19:45:45.194855 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:45.194807    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tbtp8" event={"ID":"9911a8f7-3135-4081-b2ad-c9efbf7936b7","Type":"ContainerStarted","Data":"f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b"}
Jul 28 19:45:45.196620 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:45.196592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z" event={"ID":"2c52c858-4373-4f1a-8cc3-4fc3f4037aac","Type":"ContainerStarted","Data":"a8ed645d08b5fc2e1e6444b09ecc7082c702ce18cf287f7d7af6ff8b9d7a24f5"}
Jul 28 19:45:45.196739 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:45.196680    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:45.209769 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:45.209721    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-tbtp8" podStartSLOduration=1.320965008 podStartE2EDuration="5.209702006s" totalImagesPullingTime="3.888736998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:45:45.209396704 +0000 UTC m=+563.670155304" watchObservedRunningTime="2026-07-28 19:45:45.209702006 +0000 UTC m=+563.670460605"
Jul 28 19:45:45.225882 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:45.225836    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z" podStartSLOduration=2.015168694 podStartE2EDuration="6.225822158s" totalImagesPullingTime="4.210653464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:45:45.224476854 +0000 UTC m=+563.685235457" watchObservedRunningTime="2026-07-28 19:45:45.225822158 +0000 UTC m=+563.686580781"
Jul 28 19:45:46.200706 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:46.200669    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-tbtp8" podUID="9911a8f7-3135-4081-b2ad-c9efbf7936b7" containerName="authorino" containerID="cri-o://f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b" gracePeriod=30
Jul 28 19:45:46.440991 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:46.440969    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tbtp8"
Jul 28 19:45:46.525341 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:46.525285    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxddt\" (UniqueName: \"kubernetes.io/projected/9911a8f7-3135-4081-b2ad-c9efbf7936b7-kube-api-access-sxddt\") pod \"9911a8f7-3135-4081-b2ad-c9efbf7936b7\" (UID: \"9911a8f7-3135-4081-b2ad-c9efbf7936b7\") "
Jul 28 19:45:46.527274 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:46.527252    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9911a8f7-3135-4081-b2ad-c9efbf7936b7-kube-api-access-sxddt" (OuterVolumeSpecName: "kube-api-access-sxddt") pod "9911a8f7-3135-4081-b2ad-c9efbf7936b7" (UID: "9911a8f7-3135-4081-b2ad-c9efbf7936b7"). InnerVolumeSpecName "kube-api-access-sxddt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:45:46.626180 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:46.626155    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxddt\" (UniqueName: \"kubernetes.io/projected/9911a8f7-3135-4081-b2ad-c9efbf7936b7-kube-api-access-sxddt\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 19:45:47.205631 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.205595    2555 generic.go:356] "Generic (PLEG): container finished" podID="9911a8f7-3135-4081-b2ad-c9efbf7936b7" containerID="f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b" exitCode=0
Jul 28 19:45:47.206084 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.205639    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-tbtp8"
Jul 28 19:45:47.206084 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.205657    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tbtp8" event={"ID":"9911a8f7-3135-4081-b2ad-c9efbf7936b7","Type":"ContainerDied","Data":"f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b"}
Jul 28 19:45:47.206084 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.205689    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-tbtp8" event={"ID":"9911a8f7-3135-4081-b2ad-c9efbf7936b7","Type":"ContainerDied","Data":"40a3e474f6d332c9e08c2e53bbccd0fd6d29cc8f2a591df5d7845d22d46c6ab2"}
Jul 28 19:45:47.206084 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.205708    2555 scope.go:117] "RemoveContainer" containerID="f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b"
Jul 28 19:45:47.214658 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.214631    2555 scope.go:117] "RemoveContainer" containerID="f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b"
Jul 28 19:45:47.214936 ip-10-0-138-173 kubenswrapper[2555]: E0728 19:45:47.214910    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b\": container with ID starting with f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b not found: ID does not exist" containerID="f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b"
Jul 28 19:45:47.215018 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.214949    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b"} err="failed to get container status \"f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b\": rpc error: code = NotFound desc = could not find container \"f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b\": container with ID starting with f27a3b812d12fb37153bacda2777f84e2c7e76377400c2aedc13a1f6630f693b not found: ID does not exist"
Jul 28 19:45:47.227998 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.227972    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbtp8"]
Jul 28 19:45:47.232190 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.232169    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-tbtp8"]
Jul 28 19:45:47.984144 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:47.984111    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9911a8f7-3135-4081-b2ad-c9efbf7936b7" path="/var/lib/kubelet/pods/9911a8f7-3135-4081-b2ad-c9efbf7936b7/volumes"
Jul 28 19:45:52.696217 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:52.696183    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:52.796048 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:52.796015    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:56.201713 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:56.201682    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"
Jul 28 19:45:57.482288 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:57.482254    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:57.577648 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:57.577617    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:57.947650 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:57.947614    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:45:58.377981 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:45:58.377945    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:10.431068 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:10.431035    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:10.552537 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:10.552509    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:10.968820 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:10.968793    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:17.878860 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:17.878823    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:17.977610 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:17.977578    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:18.040448 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:18.040425    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:18.121173 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:18.121148    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:21.189162 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:21.189124    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:21.289515 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:21.289482    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:45.703321 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:45.703281    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:50.360923 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:50.360877    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:50.450754 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:50.450725    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:51.017806 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:51.017773    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:51.206695 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:51.206665    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:51.644264 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:51.644234    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:51.747736 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:51.747704    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:46:51.754843 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:46:51.754815    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:47:01.474747 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:01.474712    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:01.599752 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:01.599719    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:16.832217 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:16.832181    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:17.309906 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:17.309852    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:47:31.173947 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:31.173883    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:31.248003 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:31.247967    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:46.337266 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:46.337217    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:47.100560 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:47.100518    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:47:52.990850 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.990818    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"]
Jul 28 19:47:52.991305 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.991123    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9911a8f7-3135-4081-b2ad-c9efbf7936b7" containerName="authorino"
Jul 28 19:47:52.991305 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.991136    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9911a8f7-3135-4081-b2ad-c9efbf7936b7" containerName="authorino"
Jul 28 19:47:52.991305 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.991194    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9911a8f7-3135-4081-b2ad-c9efbf7936b7" containerName="authorino"
Jul 28 19:47:52.993083 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.993067    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:52.995504 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.995477    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 19:47:52.996531 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.996473    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 19:47:52.996643 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.996588    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 19:47:52.996706 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:52.996689    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5hlzv\""
Jul 28 19:47:53.001805 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.001784    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.001913 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.001828    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.001913 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.001844    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15586622-87c4-4a53-97c8-18940ff00c50-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.002002 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.001902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.002002 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.001976    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.002075 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.002016    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lw2m\" (UniqueName: \"kubernetes.io/projected/15586622-87c4-4a53-97c8-18940ff00c50-kube-api-access-6lw2m\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.005661 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.005637    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"]
Jul 28 19:47:53.019846 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.019813    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:53.102738 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.102715    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.102840 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.102747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.102840 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.102770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lw2m\" (UniqueName: \"kubernetes.io/projected/15586622-87c4-4a53-97c8-18940ff00c50-kube-api-access-6lw2m\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.102840 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.102792    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.102840 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.102828    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.102840 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.102842    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15586622-87c4-4a53-97c8-18940ff00c50-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.103163 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.103142    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.103228 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.103182    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.103228 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.103217    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.105093 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.105074    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15586622-87c4-4a53-97c8-18940ff00c50-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.105259 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.105243    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15586622-87c4-4a53-97c8-18940ff00c50-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.113388 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.113364    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lw2m\" (UniqueName: \"kubernetes.io/projected/15586622-87c4-4a53-97c8-18940ff00c50-kube-api-access-6lw2m\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vkjqk\" (UID: \"15586622-87c4-4a53-97c8-18940ff00c50\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.302832 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.302802    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:47:53.426987 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:47:53.426959    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15586622_87c4_4a53_97c8_18940ff00c50.slice/crio-b77b82a9937e2b6332c542f872894ec201847ce0dde239dada17c2347f546ba8 WatchSource:0}: Error finding container b77b82a9937e2b6332c542f872894ec201847ce0dde239dada17c2347f546ba8: Status 404 returned error can't find the container with id b77b82a9937e2b6332c542f872894ec201847ce0dde239dada17c2347f546ba8
Jul 28 19:47:53.428402 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.428380    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 19:47:53.429937 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.429917    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"]
Jul 28 19:47:53.734358 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:53.734274    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk" event={"ID":"15586622-87c4-4a53-97c8-18940ff00c50","Type":"ContainerStarted","Data":"b77b82a9937e2b6332c542f872894ec201847ce0dde239dada17c2347f546ba8"}
Jul 28 19:47:54.201967 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:54.201937    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:47:54.740229 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:54.740152    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk" event={"ID":"15586622-87c4-4a53-97c8-18940ff00c50","Type":"ContainerStarted","Data":"a8dcaad87fd739d5eee4b1fe1f817e0d24edd09cc3b4f3c61d61dc740a23edec"}
Jul 28 19:47:54.759370 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:47:54.759324    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk" podStartSLOduration=1.7446639849999999 podStartE2EDuration="2.759309394s" totalImagesPullingTime="1.014645409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:54.758084848 +0000 UTC m=+693.218843450" watchObservedRunningTime="2026-07-28 19:47:54.759309394 +0000 UTC m=+693.220067993"
Jul 28 19:48:00.229004 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:00.228929    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:48:01.896994 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:01.896953    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:48:03.303148 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:03.303110    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:48:03.303608 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:03.303198    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:48:03.315402 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:03.315380    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:48:03.793670 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:03.793633    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vkjqk"
Jul 28 19:48:11.620482 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:11.620441    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:48:13.994025 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:13.993936    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:48:23.999606 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:23.999572    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"]
Jul 28 19:48:24.002807 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.002791    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.005259 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.005237    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 28 19:48:24.015514 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.015493    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"]
Jul 28 19:48:24.020138 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.020116    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:48:24.114383 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.114355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99347c96-f13b-4823-9b11-c32b3453fe47-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.114500 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.114393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.114500 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.114428    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.114500 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.114451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.114500 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.114481    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.114639 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.114554    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48zg\" (UniqueName: \"kubernetes.io/projected/99347c96-f13b-4823-9b11-c32b3453fe47-kube-api-access-z48zg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.215529 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.215501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99347c96-f13b-4823-9b11-c32b3453fe47-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.215642 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.215535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.215642 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.215557    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.215642 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.215599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.215642 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.215638    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.215860 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.215682    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48zg\" (UniqueName: \"kubernetes.io/projected/99347c96-f13b-4823-9b11-c32b3453fe47-kube-api-access-z48zg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.215964 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.215941    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.216042 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.216019    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.216095 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.216030    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.217964 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.217941    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99347c96-f13b-4823-9b11-c32b3453fe47-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.218310 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.218289    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99347c96-f13b-4823-9b11-c32b3453fe47-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.226281 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.226256    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48zg\" (UniqueName: \"kubernetes.io/projected/99347c96-f13b-4823-9b11-c32b3453fe47-kube-api-access-z48zg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p\" (UID: \"99347c96-f13b-4823-9b11-c32b3453fe47\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.311825 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.311800    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:24.439830 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.439806    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"]
Jul 28 19:48:24.442203 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:48:24.442177    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99347c96_f13b_4823_9b11_c32b3453fe47.slice/crio-95b1261bf46aa3fafa95c191a6808646f2c32c3d0cd383fbd06ed7255cb8c3e8 WatchSource:0}: Error finding container 95b1261bf46aa3fafa95c191a6808646f2c32c3d0cd383fbd06ed7255cb8c3e8: Status 404 returned error can't find the container with id 95b1261bf46aa3fafa95c191a6808646f2c32c3d0cd383fbd06ed7255cb8c3e8
Jul 28 19:48:24.870876 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.870839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p" event={"ID":"99347c96-f13b-4823-9b11-c32b3453fe47","Type":"ContainerStarted","Data":"053d6feb56af06df2df5a00aa5b575223e910e24a4c2d9ac3cab6d4ea6e656be"}
Jul 28 19:48:24.871027 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.870904    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p" event={"ID":"99347c96-f13b-4823-9b11-c32b3453fe47","Type":"ContainerStarted","Data":"95b1261bf46aa3fafa95c191a6808646f2c32c3d0cd383fbd06ed7255cb8c3e8"}
Jul 28 19:48:24.888660 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:24.888597    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p" podStartSLOduration=1.615930701 podStartE2EDuration="1.888578201s" totalImagesPullingTime="272.6475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:24.887880216 +0000 UTC m=+723.348638817" watchObservedRunningTime="2026-07-28 19:48:24.888578201 +0000 UTC m=+723.349336805"
Jul 28 19:48:26.701387 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:26.701354    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:48:34.312810 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:34.312770    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:34.313448 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:34.312827    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:34.324902 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:34.324865    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:34.924651 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:34.924621    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p"
Jul 28 19:48:36.593135 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:36.593100    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:48:36.692420 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:48:36.692389    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:49:18.591163 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.591121    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b8b8cfdc8-jc8vx"]
Jul 28 19:49:18.594411 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.594390    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx"
Jul 28 19:49:18.598311 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.598288    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grx9k\""
Jul 28 19:49:18.598415 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.598340    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 19:49:18.602664 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.602637    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b8b8cfdc8-jc8vx"]
Jul 28 19:49:18.698143 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.698117    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a411cba-bd53-4f31-bfb9-098ec849238d-tls-cert\") pod \"authorino-5b8b8cfdc8-jc8vx\" (UID: \"5a411cba-bd53-4f31-bfb9-098ec849238d\") " pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx"
Jul 28 19:49:18.698276 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.698145    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbbr\" (UniqueName: \"kubernetes.io/projected/5a411cba-bd53-4f31-bfb9-098ec849238d-kube-api-access-bjbbr\") pod \"authorino-5b8b8cfdc8-jc8vx\" (UID: \"5a411cba-bd53-4f31-bfb9-098ec849238d\") " pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx"
Jul 28 19:49:18.799023 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.798991    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a411cba-bd53-4f31-bfb9-098ec849238d-tls-cert\") pod \"authorino-5b8b8cfdc8-jc8vx\" (UID: \"5a411cba-bd53-4f31-bfb9-098ec849238d\") " pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx"
Jul 28 19:49:18.799150 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.799026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbbr\" (UniqueName: \"kubernetes.io/projected/5a411cba-bd53-4f31-bfb9-098ec849238d-kube-api-access-bjbbr\") pod \"authorino-5b8b8cfdc8-jc8vx\" (UID: \"5a411cba-bd53-4f31-bfb9-098ec849238d\") " pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx"
Jul 28 19:49:18.801597 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.801576    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5a411cba-bd53-4f31-bfb9-098ec849238d-tls-cert\") pod \"authorino-5b8b8cfdc8-jc8vx\" (UID: \"5a411cba-bd53-4f31-bfb9-098ec849238d\") " pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx"
Jul 28 19:49:18.812166 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.812141    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbbr\" (UniqueName: \"kubernetes.io/projected/5a411cba-bd53-4f31-bfb9-098ec849238d-kube-api-access-bjbbr\") pod \"authorino-5b8b8cfdc8-jc8vx\" (UID: \"5a411cba-bd53-4f31-bfb9-098ec849238d\") " pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx"
Jul 28 19:49:18.905057 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.904995    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx"
Jul 28 19:49:18.962483 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:18.962458    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:49:19.042085 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:19.042061    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b8b8cfdc8-jc8vx"]
Jul 28 19:49:19.044562 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:19.044536    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:49:19.044666 ip-10-0-138-173 kubenswrapper[2555]: W0728 19:49:19.044577    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a411cba_bd53_4f31_bfb9_098ec849238d.slice/crio-1ddb20a563df77bf0e334292ce78d3386ce6c69a4e221d3422e3505a3e4bf4f9 WatchSource:0}: Error finding container 1ddb20a563df77bf0e334292ce78d3386ce6c69a4e221d3422e3505a3e4bf4f9: Status 404 returned error can't find the container with id 1ddb20a563df77bf0e334292ce78d3386ce6c69a4e221d3422e3505a3e4bf4f9
Jul 28 19:49:19.095080 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:19.095054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx" event={"ID":"5a411cba-bd53-4f31-bfb9-098ec849238d","Type":"ContainerStarted","Data":"1ddb20a563df77bf0e334292ce78d3386ce6c69a4e221d3422e3505a3e4bf4f9"}
Jul 28 19:49:20.101258 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:20.101224    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx" event={"ID":"5a411cba-bd53-4f31-bfb9-098ec849238d","Type":"ContainerStarted","Data":"45257b3383c59742cf5cb3e2c6a5c7174eac84d46216c67e196a8addcac1621c"}
Jul 28 19:49:20.120353 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:20.120321    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:49:20.135951 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:20.135906    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b8b8cfdc8-jc8vx" podStartSLOduration=1.656326492 podStartE2EDuration="2.135875166s" totalImagesPullingTime="479.548674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:49:20.135077478 +0000 UTC m=+778.595836088" watchObservedRunningTime="2026-07-28 19:49:20.135875166 +0000 UTC m=+778.596633764"
Jul 28 19:49:20.834592 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:20.834562    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:49:45.114391 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:45.114292    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:49:56.900877 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:56.900833    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:49:59.140914 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:59.140866    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:49:59.201813 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:49:59.201780    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:50:05.710148 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:50:05.710110    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:50:17.202259 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:50:17.202220    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:51:26.488346 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:51:26.488300    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:51:29.044554 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:51:29.044515    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:51:52.298780 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:51:52.298749    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:52:04.100719 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:04.100682    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:52:09.437334 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:09.437299    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:52:11.510123 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:11.510089    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:52:15.409360 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:15.409322    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:52:15.523410 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:15.523376    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:52:15.611147 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:15.611117    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:52:16.553768 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:16.553714    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:52:21.585740 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:21.585705    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:52:21.685257 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:21.685232    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:52:38.495240 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:38.495198    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:52:48.100208 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:48.100122    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:52:56.600024 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:52:56.599976    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:53:05.905006 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:53:05.904969    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:53:16.309414 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:53:16.309376    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:53:26.004491 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:53:26.004444    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:53:37.496280 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:53:37.496208    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:53:47.003822 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:53:47.003775    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:54:00.588015 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:54:00.587979    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:54:00.676615 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:54:00.676576    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:54:09.708468 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:54:09.708428    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:54:18.596477 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:54:18.596381    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:54:43.900383 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:54:43.900350    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:54:54.104560 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:54:54.104524    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:55:01.942475 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:01.942427    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:55:03.402790 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:03.402750    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:55:20.305364 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:20.305328    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:55:31.499030 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:31.498997    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:55:40.803586 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:40.803551    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:55:45.693303 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:45.693214    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:55:46.817612 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:46.817584    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 19:55:50.297679 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:50.297643    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:55:57.408735 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:55:57.408698    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:56:06.902952 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:56:06.902903    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:58:04.695252 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:58:04.695214    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:58:15.497018 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:58:15.496984    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:58:31.014204 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:58:31.014168    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:58:45.595557 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:58:45.595480    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:58:54.494738 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:58:54.494698    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:59:03.999378 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:59:03.999333    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:59:13.591097 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:59:13.591051    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:59:22.805505 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:59:22.805472    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:59:34.900186 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:59:34.900147    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:59:44.297031 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:59:44.296997    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 19:59:54.000663 ip-10-0-138-173 kubenswrapper[2555]: I0728 19:59:54.000622    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:00:05.797411 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:05.797369    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:00:31.103094 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:31.103013    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:00:38.824424 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:38.824386    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:00:44.203983 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.203935    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"]
Jul 28 20:00:44.207530 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.207502    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.209882 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.209857    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d9e6e1f1-kserve-self-signed-certs\""
Jul 28 20:00:44.215536 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.215512    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"]
Jul 28 20:00:44.257161 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.257129    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-dshm\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.257291 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.257177    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkzlk\" (UniqueName: \"kubernetes.io/projected/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-kube-api-access-wkzlk\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.257291 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.257198    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tls-certs\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.257291 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.257224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-model-cache\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.257291 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.257274    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tmp-dir\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.257515 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.257336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-home\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.357624 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.357596    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-dshm\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.357737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.357639    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkzlk\" (UniqueName: \"kubernetes.io/projected/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-kube-api-access-wkzlk\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.357800 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.357746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tls-certs\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.357800 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.357784    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-model-cache\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.357930 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.357807    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tmp-dir\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.357930 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.357848    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-home\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.358154 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.358134    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-model-cache\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.358222 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.358160    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-home\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.358274 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.358229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tmp-dir\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.359867 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.359849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-dshm\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.360078 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.360062    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tls-certs\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.365141 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.365120    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkzlk\" (UniqueName: \"kubernetes.io/projected/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-kube-api-access-wkzlk\") pod \"e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") " pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.518446 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.518394    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:44.643825 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.643799    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"]
Jul 28 20:00:44.651137 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:00:44.650095    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d6c4eb5_ba01_4def_b81f_9ff7f9abee91.slice/crio-fdfaf06fa38a3a3672688a4a9b7674f2647bbbe61d3fc47cfeaec8fb127851f7 WatchSource:0}: Error finding container fdfaf06fa38a3a3672688a4a9b7674f2647bbbe61d3fc47cfeaec8fb127851f7: Status 404 returned error can't find the container with id fdfaf06fa38a3a3672688a4a9b7674f2647bbbe61d3fc47cfeaec8fb127851f7
Jul 28 20:00:44.997044 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.997006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw" event={"ID":"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91","Type":"ContainerStarted","Data":"bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b"}
Jul 28 20:00:44.997182 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:44.997051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw" event={"ID":"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91","Type":"ContainerStarted","Data":"fdfaf06fa38a3a3672688a4a9b7674f2647bbbe61d3fc47cfeaec8fb127851f7"}
Jul 28 20:00:45.014241 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:45.014196    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw" podStartSLOduration=1.014182437 podStartE2EDuration="1.014182437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:45.012802542 +0000 UTC m=+1463.473561141" watchObservedRunningTime="2026-07-28 20:00:45.014182437 +0000 UTC m=+1463.474941035"
Jul 28 20:00:45.801165 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:45.801131    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:00:48.802046 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:48.802010    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:00:54.518994 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:54.518958    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:54.519431 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:54.519131    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:54.531602 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:54.531576    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:55.052193 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:55.052169    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:00:59.879507 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.879453    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"]
Jul 28 20:00:59.882790 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.882769    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:00:59.885373 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.885349    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d9e6e1f1-kserve-self-signed-certs\""
Jul 28 20:00:59.892865 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.892839    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"]
Jul 28 20:00:59.969579 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.969551    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-tmp-dir\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:00:59.969692 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.969583    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/3d76ec2b-b825-404f-97a2-d776711989f1-kube-api-access-mnx4n\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:00:59.969692 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.969606    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d76ec2b-b825-404f-97a2-d776711989f1-tls-certs\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:00:59.969818 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.969759    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-dshm\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:00:59.969871 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.969813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-model-cache\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:00:59.969871 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.969843    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-home\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:00:59.991417 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:00:59.991388    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:01:00.071019 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.070992    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/3d76ec2b-b825-404f-97a2-d776711989f1-kube-api-access-mnx4n\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.071125 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.071025    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d76ec2b-b825-404f-97a2-d776711989f1-tls-certs\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.071125 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.071066    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-dshm\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.071125 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.071090    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-model-cache\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.071125 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.071106    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-home\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.071336 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.071135    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-tmp-dir\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.071559 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.071537    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-home\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.071669 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.071567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-tmp-dir\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.071669 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.071645    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-model-cache\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.073324 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.073295    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-dshm\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.073573 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.073556    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d76ec2b-b825-404f-97a2-d776711989f1-tls-certs\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.080540 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.080514    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/3d76ec2b-b825-404f-97a2-d776711989f1-kube-api-access-mnx4n\") pod \"e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") " pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.194144 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.194087    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:00.317880 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:00.317857    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"]
Jul 28 20:01:00.320255 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:01:00.320229    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d76ec2b_b825_404f_97a2_d776711989f1.slice/crio-eebf012c5fd7d7c2bb81a78374e69ddd9973bc7a16c4bcf2a384a432dc870413 WatchSource:0}: Error finding container eebf012c5fd7d7c2bb81a78374e69ddd9973bc7a16c4bcf2a384a432dc870413: Status 404 returned error can't find the container with id eebf012c5fd7d7c2bb81a78374e69ddd9973bc7a16c4bcf2a384a432dc870413
Jul 28 20:01:01.068276 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:01.068237    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55" event={"ID":"3d76ec2b-b825-404f-97a2-d776711989f1","Type":"ContainerStarted","Data":"fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54"}
Jul 28 20:01:01.068697 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:01.068290    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55" event={"ID":"3d76ec2b-b825-404f-97a2-d776711989f1","Type":"ContainerStarted","Data":"eebf012c5fd7d7c2bb81a78374e69ddd9973bc7a16c4bcf2a384a432dc870413"}
Jul 28 20:01:01.086927 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:01.086856    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55" podStartSLOduration=2.086840438 podStartE2EDuration="2.086840438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:01:01.085520292 +0000 UTC m=+1479.546278891" watchObservedRunningTime="2026-07-28 20:01:01.086840438 +0000 UTC m=+1479.547599015"
Jul 28 20:01:04.901428 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:04.901398    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:01:10.194191 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:10.194160    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:10.194480 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:10.194201    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:10.206657 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:10.206636    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:11.124932 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:11.124884    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:38.652462 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:38.652423    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"]
Jul 28 20:01:38.674855 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:38.674825    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:01:40.337710 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:40.337677    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"]
Jul 28 20:01:40.356703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:40.356670    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:01:40.392528 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:40.392500    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:01:53.672158 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.672088    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw" podUID="9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" containerName="main" containerID="cri-o://bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b" gracePeriod=45
Jul 28 20:01:53.911175 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.911152    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:01:53.963673 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.963616    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-dshm\") pod \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") "
Jul 28 20:01:53.963673 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.963663    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-home\") pod \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") "
Jul 28 20:01:53.963848 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.963685    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkzlk\" (UniqueName: \"kubernetes.io/projected/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-kube-api-access-wkzlk\") pod \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") "
Jul 28 20:01:53.963848 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.963704    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tmp-dir\") pod \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") "
Jul 28 20:01:53.963980 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.963846    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tls-certs\") pod \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") "
Jul 28 20:01:53.963980 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.963921    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-model-cache\") pod \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\" (UID: \"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91\") "
Jul 28 20:01:53.963980 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.963934    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-home" (OuterVolumeSpecName: "home") pod "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" (UID: "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:01:53.963980 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.963949    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" (UID: "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:01:53.964169 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.964148    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:53.964231 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.964173    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:53.964231 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.964195    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" (UID: "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:01:53.965810 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.965783    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-dshm" (OuterVolumeSpecName: "dshm") pod "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" (UID: "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:01:53.965914 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.965863    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" (UID: "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:01:53.965914 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:53.965865    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-kube-api-access-wkzlk" (OuterVolumeSpecName: "kube-api-access-wkzlk") pod "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" (UID: "9d6c4eb5-ba01-4def-b81f-9ff7f9abee91"). InnerVolumeSpecName "kube-api-access-wkzlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:01:54.065037 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.065013    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:54.065037 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.065035    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkzlk\" (UniqueName: \"kubernetes.io/projected/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-kube-api-access-wkzlk\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:54.065158 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.065046    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:54.065158 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.065055    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:54.293428 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.293404    2555 generic.go:356] "Generic (PLEG): container finished" podID="9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" containerID="bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b" exitCode=0
Jul 28 20:01:54.293539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.293460    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"
Jul 28 20:01:54.293539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.293482    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw" event={"ID":"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91","Type":"ContainerDied","Data":"bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b"}
Jul 28 20:01:54.293539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.293528    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw" event={"ID":"9d6c4eb5-ba01-4def-b81f-9ff7f9abee91","Type":"ContainerDied","Data":"fdfaf06fa38a3a3672688a4a9b7674f2647bbbe61d3fc47cfeaec8fb127851f7"}
Jul 28 20:01:54.293644 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.293552    2555 scope.go:117] "RemoveContainer" containerID="bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b"
Jul 28 20:01:54.301951 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.301935    2555 scope.go:117] "RemoveContainer" containerID="bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b"
Jul 28 20:01:54.302210 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:01:54.302191    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b\": container with ID starting with bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b not found: ID does not exist" containerID="bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b"
Jul 28 20:01:54.302262 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.302217    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b"} err="failed to get container status \"bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b\": rpc error: code = NotFound desc = could not find container \"bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b\": container with ID starting with bba00d123f6f06083dd8ad936ffbd484045a1e7a4cde3a88527758b40e6a5c4b not found: ID does not exist"
Jul 28 20:01:54.309792 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.309769    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"]
Jul 28 20:01:54.315663 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:54.315639    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d9e6e1f1-kserve-79bf95d7f9-qg6bw"]
Jul 28 20:01:55.360403 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.360363    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55" podUID="3d76ec2b-b825-404f-97a2-d776711989f1" containerName="main" containerID="cri-o://fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54" gracePeriod=45
Jul 28 20:01:55.597622 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.597596    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:01:55.600974 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.600954    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:55.676615 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676552    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-home\") pod \"3d76ec2b-b825-404f-97a2-d776711989f1\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") "
Jul 28 20:01:55.676732 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676637    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-model-cache\") pod \"3d76ec2b-b825-404f-97a2-d776711989f1\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") "
Jul 28 20:01:55.676732 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676721    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-tmp-dir\") pod \"3d76ec2b-b825-404f-97a2-d776711989f1\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") "
Jul 28 20:01:55.676850 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676743    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d76ec2b-b825-404f-97a2-d776711989f1-tls-certs\") pod \"3d76ec2b-b825-404f-97a2-d776711989f1\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") "
Jul 28 20:01:55.676850 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676760    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-dshm\") pod \"3d76ec2b-b825-404f-97a2-d776711989f1\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") "
Jul 28 20:01:55.676850 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676776    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d76ec2b-b825-404f-97a2-d776711989f1" (UID: "3d76ec2b-b825-404f-97a2-d776711989f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:01:55.676850 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676804    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/3d76ec2b-b825-404f-97a2-d776711989f1-kube-api-access-mnx4n\") pod \"3d76ec2b-b825-404f-97a2-d776711989f1\" (UID: \"3d76ec2b-b825-404f-97a2-d776711989f1\") "
Jul 28 20:01:55.676850 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676820    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-home" (OuterVolumeSpecName: "home") pod "3d76ec2b-b825-404f-97a2-d776711989f1" (UID: "3d76ec2b-b825-404f-97a2-d776711989f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:01:55.677130 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.676964    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d76ec2b-b825-404f-97a2-d776711989f1" (UID: "3d76ec2b-b825-404f-97a2-d776711989f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:01:55.677130 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.677089    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:55.677225 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.677170    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:55.678848 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.678828    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d76ec2b-b825-404f-97a2-d776711989f1-kube-api-access-mnx4n" (OuterVolumeSpecName: "kube-api-access-mnx4n") pod "3d76ec2b-b825-404f-97a2-d776711989f1" (UID: "3d76ec2b-b825-404f-97a2-d776711989f1"). InnerVolumeSpecName "kube-api-access-mnx4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:01:55.679188 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.679162    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-dshm" (OuterVolumeSpecName: "dshm") pod "3d76ec2b-b825-404f-97a2-d776711989f1" (UID: "3d76ec2b-b825-404f-97a2-d776711989f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:01:55.679271 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.679212    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d76ec2b-b825-404f-97a2-d776711989f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d76ec2b-b825-404f-97a2-d776711989f1" (UID: "3d76ec2b-b825-404f-97a2-d776711989f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:01:55.778183 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.778162    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnx4n\" (UniqueName: \"kubernetes.io/projected/3d76ec2b-b825-404f-97a2-d776711989f1-kube-api-access-mnx4n\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:55.778183 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.778183    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:55.778302 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.778195    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d76ec2b-b825-404f-97a2-d776711989f1-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:55.778302 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.778203    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d76ec2b-b825-404f-97a2-d776711989f1-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:01:55.983524 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:55.983457    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" path="/var/lib/kubelet/pods/9d6c4eb5-ba01-4def-b81f-9ff7f9abee91/volumes"
Jul 28 20:01:56.303549 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.303517    2555 generic.go:356] "Generic (PLEG): container finished" podID="3d76ec2b-b825-404f-97a2-d776711989f1" containerID="fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54" exitCode=0
Jul 28 20:01:56.303682 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.303562    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"
Jul 28 20:01:56.303682 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.303600    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55" event={"ID":"3d76ec2b-b825-404f-97a2-d776711989f1","Type":"ContainerDied","Data":"fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54"}
Jul 28 20:01:56.303682 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.303645    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55" event={"ID":"3d76ec2b-b825-404f-97a2-d776711989f1","Type":"ContainerDied","Data":"eebf012c5fd7d7c2bb81a78374e69ddd9973bc7a16c4bcf2a384a432dc870413"}
Jul 28 20:01:56.303682 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.303662    2555 scope.go:117] "RemoveContainer" containerID="fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54"
Jul 28 20:01:56.311383 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.311364    2555 scope.go:117] "RemoveContainer" containerID="fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54"
Jul 28 20:01:56.311661 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:01:56.311642    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54\": container with ID starting with fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54 not found: ID does not exist" containerID="fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54"
Jul 28 20:01:56.311746 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.311670    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54"} err="failed to get container status \"fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54\": rpc error: code = NotFound desc = could not find container \"fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54\": container with ID starting with fba67136f969fd433ea1fc0a9c6e18fe8eaec443901480578201588e8c4eec54 not found: ID does not exist"
Jul 28 20:01:56.323554 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.323535    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"]
Jul 28 20:01:56.329909 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:56.329876    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d9e6e1f1-kserve-6f9fd5f747-lbf55"]
Jul 28 20:01:57.983826 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:01:57.983790    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d76ec2b-b825-404f-97a2-d776711989f1" path="/var/lib/kubelet/pods/3d76ec2b-b825-404f-97a2-d776711989f1/volumes"
Jul 28 20:02:03.695853 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:02:03.695806    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:02:14.607621 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:02:14.607578    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:02:25.358856 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:02:25.358820    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:25.500221 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:02:25.500194    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:02:34.504374 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:02:34.504331    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:02:43.595926 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:02:43.595877    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:02:50.465618 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:02:50.465578    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:02:54.107851 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:02:54.107813    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:03:03.895764 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:03:03.895731    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:03:17.301228 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:03:17.301148    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:03:26.697905 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:03:26.697847    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:03:36.204295 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:03:36.204257    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:03:45.597597 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:03:45.597549    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:03:58.101324 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:03:58.101285    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:04:07.407402 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:04:07.407362    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:04:18.802845 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:04:18.802813    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:04:29.811255 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:04:29.811222    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:04:30.464651 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:04:30.464600    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:39.002326 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:04:39.002284    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:04:41.714660 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:04:41.714619    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:04:53.301289 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:04:53.301205    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:05:02.705913 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:02.705865    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:05:03.486630 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:03.486597    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:06.638260 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:06.638222    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:06.738853 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:06.738813    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:07.159329 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:07.159284    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:12.857316 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:12.857272    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:13.007196 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:13.007163    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:05:13.153536 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:13.153456    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:13.254091 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:13.254065    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:13.415276 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:13.415198    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:14.188288 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:14.188256    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:15.479721 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:15.479688    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:15.564574 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:15.564535    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:15.632541 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:15.632513    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:15.732536 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:15.732475    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:16.156453 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:16.156428    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:16.261092 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:16.261065    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:23.498748 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:23.498712    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:05:46.184577 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:46.184544    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:46.251613 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:46.251579    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:46.313278 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:46.313250    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:46.378977 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:46.378953    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:46.506929 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:46.506846    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:47.541157 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:47.541122    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:49.120023 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:49.119982    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:49.549672 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:49.549641    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:49.664427 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:49.664393    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:49.926088 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:49.926010    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:50.265173 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.265109    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:50.363852 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.363822    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:05:50.409938 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.409914    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"]
Jul 28 20:05:50.410184 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.410173    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" containerName="main"
Jul 28 20:05:50.410227 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.410185    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" containerName="main"
Jul 28 20:05:50.410227 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.410195    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d76ec2b-b825-404f-97a2-d776711989f1" containerName="main"
Jul 28 20:05:50.410227 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.410200    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d76ec2b-b825-404f-97a2-d776711989f1" containerName="main"
Jul 28 20:05:50.410314 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.410257    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d76ec2b-b825-404f-97a2-d776711989f1" containerName="main"
Jul 28 20:05:50.410314 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.410266    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6c4eb5-ba01-4def-b81f-9ff7f9abee91" containerName="main"
Jul 28 20:05:50.413076 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.413057    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:05:50.413294 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.413277    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:50.415715 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.415698    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-bdcbbca7-dockercfg-5rvfk\""
Jul 28 20:05:50.415811 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.415714    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-bdcbbca7\""
Jul 28 20:05:50.421697 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.421673    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"]
Jul 28 20:05:50.547972 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.547946    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd6c3f9-5245-4b4e-bab9-f20855290184-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v\" (UID: \"dcd6c3f9-5245-4b4e-bab9-f20855290184\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:50.548084 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.547990    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7qv\" (UniqueName: \"kubernetes.io/projected/dcd6c3f9-5245-4b4e-bab9-f20855290184-kube-api-access-fw7qv\") pod \"payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v\" (UID: \"dcd6c3f9-5245-4b4e-bab9-f20855290184\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:50.649202 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.649176    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd6c3f9-5245-4b4e-bab9-f20855290184-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v\" (UID: \"dcd6c3f9-5245-4b4e-bab9-f20855290184\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:50.649308 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.649212    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7qv\" (UniqueName: \"kubernetes.io/projected/dcd6c3f9-5245-4b4e-bab9-f20855290184-kube-api-access-fw7qv\") pod \"payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v\" (UID: \"dcd6c3f9-5245-4b4e-bab9-f20855290184\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:50.649801 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.649784    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd6c3f9-5245-4b4e-bab9-f20855290184-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v\" (UID: \"dcd6c3f9-5245-4b4e-bab9-f20855290184\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:50.657713 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.657682    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7qv\" (UniqueName: \"kubernetes.io/projected/dcd6c3f9-5245-4b4e-bab9-f20855290184-kube-api-access-fw7qv\") pod \"payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v\" (UID: \"dcd6c3f9-5245-4b4e-bab9-f20855290184\") " pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:50.723695 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.723671    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:50.847212 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.847144    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"]
Jul 28 20:05:50.849483 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:05:50.849456    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd6c3f9_5245_4b4e_bab9_f20855290184.slice/crio-e7acdb91f70aafffcc80171a5bd194fb85da849a9202cf1443ddf2f44c0f114f WatchSource:0}: Error finding container e7acdb91f70aafffcc80171a5bd194fb85da849a9202cf1443ddf2f44c0f114f: Status 404 returned error can't find the container with id e7acdb91f70aafffcc80171a5bd194fb85da849a9202cf1443ddf2f44c0f114f
Jul 28 20:05:50.851384 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:50.851368    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:05:51.287669 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:51.287577    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v" event={"ID":"dcd6c3f9-5245-4b4e-bab9-f20855290184","Type":"ContainerStarted","Data":"e7acdb91f70aafffcc80171a5bd194fb85da849a9202cf1443ddf2f44c0f114f"}
Jul 28 20:05:53.296952 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:53.296923    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v" event={"ID":"dcd6c3f9-5245-4b4e-bab9-f20855290184","Type":"ContainerStarted","Data":"a4539f4f5f878b7ceba6a8d5d9a59fa3617048b8c53cb39c1a3915b8902034bd"}
Jul 28 20:05:53.297350 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:53.297044    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:05:53.315650 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:05:53.315606    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v" podStartSLOduration=0.946627235 podStartE2EDuration="3.315582011s" totalImagesPullingTime="2.368954776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:05:53.313987566 +0000 UTC m=+1771.774746167" watchObservedRunningTime="2026-07-28 20:05:53.315582011 +0000 UTC m=+1771.776340659"
Jul 28 20:06:02.504159 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:02.504123    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:06:04.304267 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:04.304228    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:06:30.281193 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.281161    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"]
Jul 28 20:06:30.281696 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.281361    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v" podUID="dcd6c3f9-5245-4b4e-bab9-f20855290184" containerName="payload-processing" containerID="cri-o://a4539f4f5f878b7ceba6a8d5d9a59fa3617048b8c53cb39c1a3915b8902034bd" gracePeriod=30
Jul 28 20:06:30.463133 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.463102    2555 generic.go:356] "Generic (PLEG): container finished" podID="dcd6c3f9-5245-4b4e-bab9-f20855290184" containerID="a4539f4f5f878b7ceba6a8d5d9a59fa3617048b8c53cb39c1a3915b8902034bd" exitCode=0
Jul 28 20:06:30.463275 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.463174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v" event={"ID":"dcd6c3f9-5245-4b4e-bab9-f20855290184","Type":"ContainerDied","Data":"a4539f4f5f878b7ceba6a8d5d9a59fa3617048b8c53cb39c1a3915b8902034bd"}
Jul 28 20:06:30.523788 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.523769    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:06:30.534542 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.534490    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw7qv\" (UniqueName: \"kubernetes.io/projected/dcd6c3f9-5245-4b4e-bab9-f20855290184-kube-api-access-fw7qv\") pod \"dcd6c3f9-5245-4b4e-bab9-f20855290184\" (UID: \"dcd6c3f9-5245-4b4e-bab9-f20855290184\") "
Jul 28 20:06:30.534650 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.534537    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd6c3f9-5245-4b4e-bab9-f20855290184-plugins-config-volume\") pod \"dcd6c3f9-5245-4b4e-bab9-f20855290184\" (UID: \"dcd6c3f9-5245-4b4e-bab9-f20855290184\") "
Jul 28 20:06:30.534874 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.534851    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd6c3f9-5245-4b4e-bab9-f20855290184-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcd6c3f9-5245-4b4e-bab9-f20855290184" (UID: "dcd6c3f9-5245-4b4e-bab9-f20855290184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:06:30.536755 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.536727    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd6c3f9-5245-4b4e-bab9-f20855290184-kube-api-access-fw7qv" (OuterVolumeSpecName: "kube-api-access-fw7qv") pod "dcd6c3f9-5245-4b4e-bab9-f20855290184" (UID: "dcd6c3f9-5245-4b4e-bab9-f20855290184"). InnerVolumeSpecName "kube-api-access-fw7qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:06:30.635711 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.635683    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7qv\" (UniqueName: \"kubernetes.io/projected/dcd6c3f9-5245-4b4e-bab9-f20855290184-kube-api-access-fw7qv\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:06:30.635711 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:30.635707    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcd6c3f9-5245-4b4e-bab9-f20855290184-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:06:31.139550 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.139512    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:31.238972 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.238914    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:31.392449 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.392362    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:31.468307 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.468285    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"
Jul 28 20:06:31.468436 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.468279    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v" event={"ID":"dcd6c3f9-5245-4b4e-bab9-f20855290184","Type":"ContainerDied","Data":"e7acdb91f70aafffcc80171a5bd194fb85da849a9202cf1443ddf2f44c0f114f"}
Jul 28 20:06:31.468436 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.468419    2555 scope.go:117] "RemoveContainer" containerID="a4539f4f5f878b7ceba6a8d5d9a59fa3617048b8c53cb39c1a3915b8902034bd"
Jul 28 20:06:31.499168 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.499146    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"]
Jul 28 20:06:31.502750 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.502731    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bdcbbca7-5db9bff89c-xxd9v"]
Jul 28 20:06:31.517820 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.517787    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:31.984845 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:31.984814    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcd6c3f9-5245-4b4e-bab9-f20855290184" path="/var/lib/kubelet/pods/dcd6c3f9-5245-4b4e-bab9-f20855290184/volumes"
Jul 28 20:06:33.857556 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:33.857505    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:33.947281 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:33.947249    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:34.010387 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.010358    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"]
Jul 28 20:06:34.010643 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.010632    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcd6c3f9-5245-4b4e-bab9-f20855290184" containerName="payload-processing"
Jul 28 20:06:34.010684 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.010645    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcd6c3f9-5245-4b4e-bab9-f20855290184" containerName="payload-processing"
Jul 28 20:06:34.010716 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.010690    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcd6c3f9-5245-4b4e-bab9-f20855290184" containerName="payload-processing"
Jul 28 20:06:34.014960 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.014941    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.017546 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.017526    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae3e83fa-serving-cert\""
Jul 28 20:06:34.017642 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.017579    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xsbnd\""
Jul 28 20:06:34.017642 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.017591    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:06:34.017642 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.017599    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:06:34.025332 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.025312    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"]
Jul 28 20:06:34.043406 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.043337    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:34.061573 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.061553    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194faa84-7c17-4400-85b2-ec7ca4f52a45-openshift-service-ca\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.061680 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.061592    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjjww\" (UniqueName: \"kubernetes.io/projected/194faa84-7c17-4400-85b2-ec7ca4f52a45-kube-api-access-vjjww\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.061680 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.061653    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194faa84-7c17-4400-85b2-ec7ca4f52a45-maas-api-tls\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.160363 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.160282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:34.162338 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.162319    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194faa84-7c17-4400-85b2-ec7ca4f52a45-openshift-service-ca\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.162420 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.162355    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjjww\" (UniqueName: \"kubernetes.io/projected/194faa84-7c17-4400-85b2-ec7ca4f52a45-kube-api-access-vjjww\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.162420 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.162389    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194faa84-7c17-4400-85b2-ec7ca4f52a45-maas-api-tls\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.162972 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.162950    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194faa84-7c17-4400-85b2-ec7ca4f52a45-openshift-service-ca\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.164725 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.164708    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194faa84-7c17-4400-85b2-ec7ca4f52a45-maas-api-tls\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.178247 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.178221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjjww\" (UniqueName: \"kubernetes.io/projected/194faa84-7c17-4400-85b2-ec7ca4f52a45-kube-api-access-vjjww\") pod \"maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.326300 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.326275    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:34.411752 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.411627    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:06:34.457650 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.457629    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"]
Jul 28 20:06:34.460062 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:06:34.460034    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194faa84_7c17_4400_85b2_ec7ca4f52a45.slice/crio-7d370545ed318848bd689741e44c07666e2aeddaef010bc467e3ddffa7572425 WatchSource:0}: Error finding container 7d370545ed318848bd689741e44c07666e2aeddaef010bc467e3ddffa7572425: Status 404 returned error can't find the container with id 7d370545ed318848bd689741e44c07666e2aeddaef010bc467e3ddffa7572425
Jul 28 20:06:34.482445 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.482414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d" event={"ID":"194faa84-7c17-4400-85b2-ec7ca4f52a45","Type":"ContainerStarted","Data":"7d370545ed318848bd689741e44c07666e2aeddaef010bc467e3ddffa7572425"}
Jul 28 20:06:34.636694 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.636669    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:34.721845 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.721782    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:06:34.773392 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.773370    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"]
Jul 28 20:06:34.777647 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.777632    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:34.780250 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.780231    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ae3e83fa\""
Jul 28 20:06:34.780331 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.780232    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ae3e83fa-dockercfg-dzxx5\""
Jul 28 20:06:34.785756 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.785737    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"]
Jul 28 20:06:34.867245 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.867224    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66be900c-7e1d-4cd7-904e-c3136c2d91e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8\" (UID: \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:34.867552 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.867288    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xvm\" (UniqueName: \"kubernetes.io/projected/66be900c-7e1d-4cd7-904e-c3136c2d91e6-kube-api-access-c4xvm\") pod \"payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8\" (UID: \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:34.968410 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.968388    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xvm\" (UniqueName: \"kubernetes.io/projected/66be900c-7e1d-4cd7-904e-c3136c2d91e6-kube-api-access-c4xvm\") pod \"payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8\" (UID: \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:34.968522 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.968465    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66be900c-7e1d-4cd7-904e-c3136c2d91e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8\" (UID: \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:34.969104 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.969085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66be900c-7e1d-4cd7-904e-c3136c2d91e6-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8\" (UID: \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:34.981141 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:34.981093    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xvm\" (UniqueName: \"kubernetes.io/projected/66be900c-7e1d-4cd7-904e-c3136c2d91e6-kube-api-access-c4xvm\") pod \"payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8\" (UID: \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\") " pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:35.089257 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:35.089229    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:35.240607 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:35.240428    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"]
Jul 28 20:06:35.243824 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:06:35.243790    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66be900c_7e1d_4cd7_904e_c3136c2d91e6.slice/crio-a4a621578a669ca247f1abb371e6e9e7b725b7850c6f93f46b15a84b9362cc3b WatchSource:0}: Error finding container a4a621578a669ca247f1abb371e6e9e7b725b7850c6f93f46b15a84b9362cc3b: Status 404 returned error can't find the container with id a4a621578a669ca247f1abb371e6e9e7b725b7850c6f93f46b15a84b9362cc3b
Jul 28 20:06:35.489808 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:35.489330    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8" event={"ID":"66be900c-7e1d-4cd7-904e-c3136c2d91e6","Type":"ContainerStarted","Data":"285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441"}
Jul 28 20:06:35.489808 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:35.489391    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8" event={"ID":"66be900c-7e1d-4cd7-904e-c3136c2d91e6","Type":"ContainerStarted","Data":"a4a621578a669ca247f1abb371e6e9e7b725b7850c6f93f46b15a84b9362cc3b"}
Jul 28 20:06:35.489808 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:35.489655    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:06:35.507080 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:35.506984    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8" podStartSLOduration=1.506965737 podStartE2EDuration="1.506965737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:35.506071819 +0000 UTC m=+1813.966830419" watchObservedRunningTime="2026-07-28 20:06:35.506965737 +0000 UTC m=+1813.967724338"
Jul 28 20:06:37.499525 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:37.499492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d" event={"ID":"194faa84-7c17-4400-85b2-ec7ca4f52a45","Type":"ContainerStarted","Data":"7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0"}
Jul 28 20:06:37.499884 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:37.499614    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:37.516854 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:37.516815    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d" podStartSLOduration=2.246256453 podStartE2EDuration="4.516804765s" totalImagesPullingTime="2.270548312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:06:37.516534186 +0000 UTC m=+1815.977292797" watchObservedRunningTime="2026-07-28 20:06:37.516804765 +0000 UTC m=+1815.977563342"
Jul 28 20:06:43.509321 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:43.509290    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:06:44.707347 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:44.707307    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:06:46.497576 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:06:46.497540    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:07:04.781587 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:04.781549    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"]
Jul 28 20:07:04.784221 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:04.781791    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d" podUID="194faa84-7c17-4400-85b2-ec7ca4f52a45" containerName="maas-api" containerID="cri-o://7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0" gracePeriod=30
Jul 28 20:07:04.810060 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:04.810026    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:04.836939 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:04.836911    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"]
Jul 28 20:07:04.837228 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:04.837203    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8" podUID="66be900c-7e1d-4cd7-904e-c3136c2d91e6" containerName="payload-processing" containerID="cri-o://285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441" gracePeriod=30
Jul 28 20:07:04.909677 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:04.909649    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:04.976079 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:04.976012    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:05.070624 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.070602    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:07:05.103819 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.103799    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:07:05.176620 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.176595    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4xvm\" (UniqueName: \"kubernetes.io/projected/66be900c-7e1d-4cd7-904e-c3136c2d91e6-kube-api-access-c4xvm\") pod \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\" (UID: \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\") "
Jul 28 20:07:05.176727 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.176635    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194faa84-7c17-4400-85b2-ec7ca4f52a45-openshift-service-ca\") pod \"194faa84-7c17-4400-85b2-ec7ca4f52a45\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") "
Jul 28 20:07:05.176727 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.176679    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194faa84-7c17-4400-85b2-ec7ca4f52a45-maas-api-tls\") pod \"194faa84-7c17-4400-85b2-ec7ca4f52a45\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") "
Jul 28 20:07:05.176837 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.176733    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjjww\" (UniqueName: \"kubernetes.io/projected/194faa84-7c17-4400-85b2-ec7ca4f52a45-kube-api-access-vjjww\") pod \"194faa84-7c17-4400-85b2-ec7ca4f52a45\" (UID: \"194faa84-7c17-4400-85b2-ec7ca4f52a45\") "
Jul 28 20:07:05.176837 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.176757    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66be900c-7e1d-4cd7-904e-c3136c2d91e6-plugins-config-volume\") pod \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\" (UID: \"66be900c-7e1d-4cd7-904e-c3136c2d91e6\") "
Jul 28 20:07:05.177110 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.177086    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/194faa84-7c17-4400-85b2-ec7ca4f52a45-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "194faa84-7c17-4400-85b2-ec7ca4f52a45" (UID: "194faa84-7c17-4400-85b2-ec7ca4f52a45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:07:05.177209 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.177188    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66be900c-7e1d-4cd7-904e-c3136c2d91e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66be900c-7e1d-4cd7-904e-c3136c2d91e6" (UID: "66be900c-7e1d-4cd7-904e-c3136c2d91e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:07:05.178753 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.178733    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66be900c-7e1d-4cd7-904e-c3136c2d91e6-kube-api-access-c4xvm" (OuterVolumeSpecName: "kube-api-access-c4xvm") pod "66be900c-7e1d-4cd7-904e-c3136c2d91e6" (UID: "66be900c-7e1d-4cd7-904e-c3136c2d91e6"). InnerVolumeSpecName "kube-api-access-c4xvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:07:05.178843 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.178826    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/194faa84-7c17-4400-85b2-ec7ca4f52a45-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "194faa84-7c17-4400-85b2-ec7ca4f52a45" (UID: "194faa84-7c17-4400-85b2-ec7ca4f52a45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:07:05.178920 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.178884    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/194faa84-7c17-4400-85b2-ec7ca4f52a45-kube-api-access-vjjww" (OuterVolumeSpecName: "kube-api-access-vjjww") pod "194faa84-7c17-4400-85b2-ec7ca4f52a45" (UID: "194faa84-7c17-4400-85b2-ec7ca4f52a45"). InnerVolumeSpecName "kube-api-access-vjjww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:07:05.277990 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.277953    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xvm\" (UniqueName: \"kubernetes.io/projected/66be900c-7e1d-4cd7-904e-c3136c2d91e6-kube-api-access-c4xvm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:05.277990 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.277987    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/194faa84-7c17-4400-85b2-ec7ca4f52a45-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:05.278131 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.278002    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/194faa84-7c17-4400-85b2-ec7ca4f52a45-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:05.278131 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.278015    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjjww\" (UniqueName: \"kubernetes.io/projected/194faa84-7c17-4400-85b2-ec7ca4f52a45-kube-api-access-vjjww\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:05.278131 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.278027    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66be900c-7e1d-4cd7-904e-c3136c2d91e6-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:05.626502 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.626466    2555 generic.go:356] "Generic (PLEG): container finished" podID="194faa84-7c17-4400-85b2-ec7ca4f52a45" containerID="7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0" exitCode=0
Jul 28 20:07:05.626606 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.626529    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"
Jul 28 20:07:05.626606 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.626562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d" event={"ID":"194faa84-7c17-4400-85b2-ec7ca4f52a45","Type":"ContainerDied","Data":"7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0"}
Jul 28 20:07:05.626709 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.626627    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d" event={"ID":"194faa84-7c17-4400-85b2-ec7ca4f52a45","Type":"ContainerDied","Data":"7d370545ed318848bd689741e44c07666e2aeddaef010bc467e3ddffa7572425"}
Jul 28 20:07:05.626709 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.626658    2555 scope.go:117] "RemoveContainer" containerID="7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0"
Jul 28 20:07:05.628998 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.628965    2555 generic.go:356] "Generic (PLEG): container finished" podID="66be900c-7e1d-4cd7-904e-c3136c2d91e6" containerID="285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441" exitCode=0
Jul 28 20:07:05.629134 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.629034    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"
Jul 28 20:07:05.629134 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.629054    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8" event={"ID":"66be900c-7e1d-4cd7-904e-c3136c2d91e6","Type":"ContainerDied","Data":"285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441"}
Jul 28 20:07:05.629254 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.629154    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8" event={"ID":"66be900c-7e1d-4cd7-904e-c3136c2d91e6","Type":"ContainerDied","Data":"a4a621578a669ca247f1abb371e6e9e7b725b7850c6f93f46b15a84b9362cc3b"}
Jul 28 20:07:05.636862 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.636836    2555 scope.go:117] "RemoveContainer" containerID="7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0"
Jul 28 20:07:05.637260 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:07:05.637239    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0\": container with ID starting with 7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0 not found: ID does not exist" containerID="7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0"
Jul 28 20:07:05.637361 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.637273    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0"} err="failed to get container status \"7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0\": rpc error: code = NotFound desc = could not find container \"7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0\": container with ID starting with 7afb92015345457a72b723732d47fa23f9887f83c2c779623d6fa13d7c59b7f0 not found: ID does not exist"
Jul 28 20:07:05.637361 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.637298    2555 scope.go:117] "RemoveContainer" containerID="285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441"
Jul 28 20:07:05.645509 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.645490    2555 scope.go:117] "RemoveContainer" containerID="285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441"
Jul 28 20:07:05.645796 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:07:05.645776    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441\": container with ID starting with 285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441 not found: ID does not exist" containerID="285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441"
Jul 28 20:07:05.645850 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.645803    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441"} err="failed to get container status \"285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441\": rpc error: code = NotFound desc = could not find container \"285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441\": container with ID starting with 285ac1d4897b0e23d1b5f83474f3c0fe04a9f0b4493fdb2b0f08a1d1e72b6441 not found: ID does not exist"
Jul 28 20:07:05.658777 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.652648    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"]
Jul 28 20:07:05.660761 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.660736    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae3e83fa-854cf67fd4-52p9d"]
Jul 28 20:07:05.674696 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.674677    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"]
Jul 28 20:07:05.678956 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.678935    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae3e83fa-5848c76988-f6hg8"]
Jul 28 20:07:05.984254 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.984192    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="194faa84-7c17-4400-85b2-ec7ca4f52a45" path="/var/lib/kubelet/pods/194faa84-7c17-4400-85b2-ec7ca4f52a45/volumes"
Jul 28 20:07:05.984550 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:05.984520    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66be900c-7e1d-4cd7-904e-c3136c2d91e6" path="/var/lib/kubelet/pods/66be900c-7e1d-4cd7-904e-c3136c2d91e6/volumes"
Jul 28 20:07:06.078828 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:06.078805    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:06.294707 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:06.294681    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:06.514703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:06.514667    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:06.825299 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:06.825272    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:06.973477 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:06.973446    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:08.698811 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:08.698770    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:07:09.084322 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.084287    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:09.171317 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.171284    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"]
Jul 28 20:07:09.171737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.171712    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66be900c-7e1d-4cd7-904e-c3136c2d91e6" containerName="payload-processing"
Jul 28 20:07:09.171737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.171733    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="66be900c-7e1d-4cd7-904e-c3136c2d91e6" containerName="payload-processing"
Jul 28 20:07:09.171950 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.171761    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="194faa84-7c17-4400-85b2-ec7ca4f52a45" containerName="maas-api"
Jul 28 20:07:09.171950 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.171771    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="194faa84-7c17-4400-85b2-ec7ca4f52a45" containerName="maas-api"
Jul 28 20:07:09.171950 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.171842    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="194faa84-7c17-4400-85b2-ec7ca4f52a45" containerName="maas-api"
Jul 28 20:07:09.171950 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.171854    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="66be900c-7e1d-4cd7-904e-c3136c2d91e6" containerName="payload-processing"
Jul 28 20:07:09.172172 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.172152    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:09.176855 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.176835    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.179364 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.179339    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-19b71b42-serving-cert\""
Jul 28 20:07:09.179487 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.179377    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:07:09.180303 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.180288    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xsbnd\""
Jul 28 20:07:09.180415 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.180339    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:07:09.186521 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.186499    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"]
Jul 28 20:07:09.239247 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.239213    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:09.309378 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.309353    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21ff7af-de9d-4580-9f0b-af793e45711a-maas-api-tls\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.309481 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.309405    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpj2\" (UniqueName: \"kubernetes.io/projected/d21ff7af-de9d-4580-9f0b-af793e45711a-kube-api-access-2lpj2\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.309481 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.309451    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21ff7af-de9d-4580-9f0b-af793e45711a-openshift-service-ca\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.410507 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.410450    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21ff7af-de9d-4580-9f0b-af793e45711a-maas-api-tls\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.410507 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.410490    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpj2\" (UniqueName: \"kubernetes.io/projected/d21ff7af-de9d-4580-9f0b-af793e45711a-kube-api-access-2lpj2\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.410639 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.410514    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21ff7af-de9d-4580-9f0b-af793e45711a-openshift-service-ca\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.411117 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.411101    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21ff7af-de9d-4580-9f0b-af793e45711a-openshift-service-ca\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.412729 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.412711    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21ff7af-de9d-4580-9f0b-af793e45711a-maas-api-tls\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.418535 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.418517    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpj2\" (UniqueName: \"kubernetes.io/projected/d21ff7af-de9d-4580-9f0b-af793e45711a-kube-api-access-2lpj2\") pod \"maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.488535 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.488513    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:09.614315 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.614291    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"]
Jul 28 20:07:09.616384 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:07:09.616355    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21ff7af_de9d_4580_9f0b_af793e45711a.slice/crio-6908836a0dc400a1102c37a2cc5cce16f4bcefa03649f009d523dff637bd3cbf WatchSource:0}: Error finding container 6908836a0dc400a1102c37a2cc5cce16f4bcefa03649f009d523dff637bd3cbf: Status 404 returned error can't find the container with id 6908836a0dc400a1102c37a2cc5cce16f4bcefa03649f009d523dff637bd3cbf
Jul 28 20:07:09.648134 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.648105    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2" event={"ID":"d21ff7af-de9d-4580-9f0b-af793e45711a","Type":"ContainerStarted","Data":"6908836a0dc400a1102c37a2cc5cce16f4bcefa03649f009d523dff637bd3cbf"}
Jul 28 20:07:09.770974 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.770911    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:09.859155 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.859128    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:09.919109 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.919086    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"]
Jul 28 20:07:09.921729 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.921714    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:09.924348 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.924319    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-19b71b42-dockercfg-pb88r\""
Jul 28 20:07:09.924830 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.924811    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-19b71b42\""
Jul 28 20:07:09.934136 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:09.934115    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"]
Jul 28 20:07:10.014969 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.014951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s7kv\" (UniqueName: \"kubernetes.io/projected/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-kube-api-access-2s7kv\") pod \"payload-processing-e2e-derive-19b71b42-747fd897d-78s2m\" (UID: \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\") " pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:10.015046 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.014988    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-19b71b42-747fd897d-78s2m\" (UID: \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\") " pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:10.115757 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.115736    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s7kv\" (UniqueName: \"kubernetes.io/projected/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-kube-api-access-2s7kv\") pod \"payload-processing-e2e-derive-19b71b42-747fd897d-78s2m\" (UID: \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\") " pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:10.115874 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.115777    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-19b71b42-747fd897d-78s2m\" (UID: \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\") " pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:10.116401 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.116380    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-plugins-config-volume\") pod \"payload-processing-e2e-derive-19b71b42-747fd897d-78s2m\" (UID: \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\") " pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:10.125763 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.125742    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s7kv\" (UniqueName: \"kubernetes.io/projected/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-kube-api-access-2s7kv\") pod \"payload-processing-e2e-derive-19b71b42-747fd897d-78s2m\" (UID: \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\") " pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:10.231923 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.231900    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:10.370998 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.370971    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"]
Jul 28 20:07:10.374231 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:07:10.374208    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b14f03_fa6d_467a_bdaf_f93dde4ed4cf.slice/crio-b3bf5d62fbddc71c2e25aa64c7b412ce528a39b5586f21ca37f9db1c74b2d567 WatchSource:0}: Error finding container b3bf5d62fbddc71c2e25aa64c7b412ce528a39b5586f21ca37f9db1c74b2d567: Status 404 returned error can't find the container with id b3bf5d62fbddc71c2e25aa64c7b412ce528a39b5586f21ca37f9db1c74b2d567
Jul 28 20:07:10.659540 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.659503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2" event={"ID":"d21ff7af-de9d-4580-9f0b-af793e45711a","Type":"ContainerStarted","Data":"0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1"}
Jul 28 20:07:10.659718 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.659603    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:10.661142 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.661115    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m" event={"ID":"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf","Type":"ContainerStarted","Data":"6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404"}
Jul 28 20:07:10.661269 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.661163    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m" event={"ID":"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf","Type":"ContainerStarted","Data":"b3bf5d62fbddc71c2e25aa64c7b412ce528a39b5586f21ca37f9db1c74b2d567"}
Jul 28 20:07:10.661269 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.661207    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:10.679380 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.679339    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2" podStartSLOduration=1.29333924 podStartE2EDuration="1.679327184s" totalImagesPullingTime="385.987944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:10.677130049 +0000 UTC m=+1849.137888647" watchObservedRunningTime="2026-07-28 20:07:10.679327184 +0000 UTC m=+1849.140085783"
Jul 28 20:07:10.696324 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:10.696287    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m" podStartSLOduration=1.6962751919999999 podStartE2EDuration="1.696275192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:07:10.695006761 +0000 UTC m=+1849.155765359" watchObservedRunningTime="2026-07-28 20:07:10.696275192 +0000 UTC m=+1849.157033811"
Jul 28 20:07:16.672708 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:16.672675    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:19.005457 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:19.005410    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:07:21.668473 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:21.668436    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:39.725494 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:39.725459    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"]
Jul 28 20:07:39.726038 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:39.725728    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2" podUID="d21ff7af-de9d-4580-9f0b-af793e45711a" containerName="maas-api" containerID="cri-o://0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1" gracePeriod=30
Jul 28 20:07:39.756531 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:39.756509    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:39.786869 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:39.786749    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"]
Jul 28 20:07:39.787028 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:39.786998    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m" podUID="54b14f03-fa6d-467a-bdaf-f93dde4ed4cf" containerName="payload-processing" containerID="cri-o://6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404" gracePeriod=30
Jul 28 20:07:39.854168 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:39.854140    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:39.917010 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:39.916972    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:39.969768 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:39.969740    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:40.047359 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.047339    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:40.049486 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.049467    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:40.075974 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.075954    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:40.129494 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.129470    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lpj2\" (UniqueName: \"kubernetes.io/projected/d21ff7af-de9d-4580-9f0b-af793e45711a-kube-api-access-2lpj2\") pod \"d21ff7af-de9d-4580-9f0b-af793e45711a\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") "
Jul 28 20:07:40.129600 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.129496    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s7kv\" (UniqueName: \"kubernetes.io/projected/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-kube-api-access-2s7kv\") pod \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\" (UID: \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\") "
Jul 28 20:07:40.129600 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.129515    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21ff7af-de9d-4580-9f0b-af793e45711a-openshift-service-ca\") pod \"d21ff7af-de9d-4580-9f0b-af793e45711a\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") "
Jul 28 20:07:40.129600 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.129590    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21ff7af-de9d-4580-9f0b-af793e45711a-maas-api-tls\") pod \"d21ff7af-de9d-4580-9f0b-af793e45711a\" (UID: \"d21ff7af-de9d-4580-9f0b-af793e45711a\") "
Jul 28 20:07:40.129767 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.129610    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-plugins-config-volume\") pod \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\" (UID: \"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf\") "
Jul 28 20:07:40.129985 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.129943    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21ff7af-de9d-4580-9f0b-af793e45711a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d21ff7af-de9d-4580-9f0b-af793e45711a" (UID: "d21ff7af-de9d-4580-9f0b-af793e45711a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:07:40.130098 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.130049    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54b14f03-fa6d-467a-bdaf-f93dde4ed4cf" (UID: "54b14f03-fa6d-467a-bdaf-f93dde4ed4cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:07:40.131592 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.131570    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21ff7af-de9d-4580-9f0b-af793e45711a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d21ff7af-de9d-4580-9f0b-af793e45711a" (UID: "d21ff7af-de9d-4580-9f0b-af793e45711a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:07:40.132087 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.132073    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-kube-api-access-2s7kv" (OuterVolumeSpecName: "kube-api-access-2s7kv") pod "54b14f03-fa6d-467a-bdaf-f93dde4ed4cf" (UID: "54b14f03-fa6d-467a-bdaf-f93dde4ed4cf"). InnerVolumeSpecName "kube-api-access-2s7kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:07:40.132161 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.132091    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21ff7af-de9d-4580-9f0b-af793e45711a-kube-api-access-2lpj2" (OuterVolumeSpecName: "kube-api-access-2lpj2") pod "d21ff7af-de9d-4580-9f0b-af793e45711a" (UID: "d21ff7af-de9d-4580-9f0b-af793e45711a"). InnerVolumeSpecName "kube-api-access-2lpj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:07:40.231007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.230956    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d21ff7af-de9d-4580-9f0b-af793e45711a-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:40.231007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.230975    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:40.231007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.230984    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lpj2\" (UniqueName: \"kubernetes.io/projected/d21ff7af-de9d-4580-9f0b-af793e45711a-kube-api-access-2lpj2\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:40.231007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.230993    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s7kv\" (UniqueName: \"kubernetes.io/projected/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf-kube-api-access-2s7kv\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:40.231007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.231001    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d21ff7af-de9d-4580-9f0b-af793e45711a-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:07:40.281469 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.281433    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:40.731645 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.731614    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:40.797787 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.797756    2555 generic.go:356] "Generic (PLEG): container finished" podID="d21ff7af-de9d-4580-9f0b-af793e45711a" containerID="0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1" exitCode=0
Jul 28 20:07:40.797922 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.797799    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"
Jul 28 20:07:40.797922 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.797838    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2" event={"ID":"d21ff7af-de9d-4580-9f0b-af793e45711a","Type":"ContainerDied","Data":"0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1"}
Jul 28 20:07:40.797922 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.797881    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2" event={"ID":"d21ff7af-de9d-4580-9f0b-af793e45711a","Type":"ContainerDied","Data":"6908836a0dc400a1102c37a2cc5cce16f4bcefa03649f009d523dff637bd3cbf"}
Jul 28 20:07:40.797922 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.797918    2555 scope.go:117] "RemoveContainer" containerID="0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1"
Jul 28 20:07:40.799285 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.799265    2555 generic.go:356] "Generic (PLEG): container finished" podID="54b14f03-fa6d-467a-bdaf-f93dde4ed4cf" containerID="6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404" exitCode=0
Jul 28 20:07:40.799349 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.799328    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"
Jul 28 20:07:40.799349 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.799341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m" event={"ID":"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf","Type":"ContainerDied","Data":"6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404"}
Jul 28 20:07:40.799448 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.799371    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m" event={"ID":"54b14f03-fa6d-467a-bdaf-f93dde4ed4cf","Type":"ContainerDied","Data":"b3bf5d62fbddc71c2e25aa64c7b412ce528a39b5586f21ca37f9db1c74b2d567"}
Jul 28 20:07:40.806104 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.806087    2555 scope.go:117] "RemoveContainer" containerID="0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1"
Jul 28 20:07:40.806354 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:07:40.806326    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1\": container with ID starting with 0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1 not found: ID does not exist" containerID="0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1"
Jul 28 20:07:40.806412 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.806353    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1"} err="failed to get container status \"0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1\": rpc error: code = NotFound desc = could not find container \"0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1\": container with ID starting with 0915ed07e7e3ea0b42a12106efc7754322b9d85d01c9a90482a2e8dc342e70a1 not found: ID does not exist"
Jul 28 20:07:40.806412 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.806371    2555 scope.go:117] "RemoveContainer" containerID="6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404"
Jul 28 20:07:40.813848 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.813826    2555 scope.go:117] "RemoveContainer" containerID="6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404"
Jul 28 20:07:40.814093 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:07:40.814075    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404\": container with ID starting with 6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404 not found: ID does not exist" containerID="6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404"
Jul 28 20:07:40.814163 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.814098    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404"} err="failed to get container status \"6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404\": rpc error: code = NotFound desc = could not find container \"6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404\": container with ID starting with 6094d025b0568b297ff4fc7b18b62d0723fe1ebe3e430bc3ace84d80fc9da404 not found: ID does not exist"
Jul 28 20:07:40.822194 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.822174    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"]
Jul 28 20:07:40.826854 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.826833    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-19b71b42-747fd897d-78s2m"]
Jul 28 20:07:40.826982 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.826965    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:40.837168 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.837150    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"]
Jul 28 20:07:40.840939 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.840922    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-19b71b42-5cdc6bb45f-d94c2"]
Jul 28 20:07:40.900518 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:40.900501    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:41.984258 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:41.984220    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54b14f03-fa6d-467a-bdaf-f93dde4ed4cf" path="/var/lib/kubelet/pods/54b14f03-fa6d-467a-bdaf-f93dde4ed4cf/volumes"
Jul 28 20:07:41.984684 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:41.984606    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d21ff7af-de9d-4580-9f0b-af793e45711a" path="/var/lib/kubelet/pods/d21ff7af-de9d-4580-9f0b-af793e45711a/volumes"
Jul 28 20:07:42.330568 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:42.330536    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:43.702804 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:43.702769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:07:48.232401 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:48.232371    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:48.338394 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:48.338369    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:48.433211 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:48.433176    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:48.538369 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:48.538299    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:48.981348 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:48.981322    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:07:53.689683 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:07:53.689645    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:08:14.891904 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:14.891850    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:20.875117 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:20.875076    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:20.990807 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:20.990770    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:21.079172 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:21.079135    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:30.546312 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:30.546271    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:30.645971 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:30.645938    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:36.953849 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:36.953808    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:42.835101 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:42.835068    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:42.900428 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:42.900393    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:42.989741 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:42.989716    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:43.087509 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:43.087441    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:53.584199 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:53.584163    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:08:54.361547 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:08:54.361516    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:00.838978 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:00.838941    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:00.991409 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:00.991372    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:01.092062 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:01.091982    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:01.188839 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:01.188795    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:03.400050 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:03.400014    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:09:12.617871 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:12.617841    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:12.716815 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:12.716745    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:12.814749 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:12.814720    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:14.470979 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:14.470903    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:15.333340 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:15.333305    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:09:20.618551 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:20.618518    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:20.718538 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:20.718498    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:21.292320 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:21.292283    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:25.899292 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:25.899257    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:09:27.076287 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:27.076254    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.176341 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:27.176317    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.320001 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:27.319974    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.628172 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:27.628139    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:27.917250 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:27.917179    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:30.400147 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.400112    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:30.500305 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.500276    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:30.626678 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.626646    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"]
Jul 28 20:09:30.627062 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.627042    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54b14f03-fa6d-467a-bdaf-f93dde4ed4cf" containerName="payload-processing"
Jul 28 20:09:30.627062 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.627063    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="54b14f03-fa6d-467a-bdaf-f93dde4ed4cf" containerName="payload-processing"
Jul 28 20:09:30.627236 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.627075    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21ff7af-de9d-4580-9f0b-af793e45711a" containerName="maas-api"
Jul 28 20:09:30.627236 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.627083    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21ff7af-de9d-4580-9f0b-af793e45711a" containerName="maas-api"
Jul 28 20:09:30.627236 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.627163    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="54b14f03-fa6d-467a-bdaf-f93dde4ed4cf" containerName="payload-processing"
Jul 28 20:09:30.627236 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.627178    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21ff7af-de9d-4580-9f0b-af793e45711a" containerName="maas-api"
Jul 28 20:09:30.630272 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.630251    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:30.633085 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.633063    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:09:30.634178 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.634155    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:09:30.634272 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.634177    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xsbnd\""
Jul 28 20:09:30.634272 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.634167    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9696d332-serving-cert\""
Jul 28 20:09:30.643818 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.643790    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"]
Jul 28 20:09:30.669515 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.669447    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:30.722607 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.722575    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:30.722734 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.722623    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-openshift-service-ca\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:30.722734 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.722683    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xm5\" (UniqueName: \"kubernetes.io/projected/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-kube-api-access-m7xm5\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:30.823454 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.823410    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:30.823454 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.823449    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-openshift-service-ca\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:30.823702 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.823493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xm5\" (UniqueName: \"kubernetes.io/projected/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-kube-api-access-m7xm5\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:30.823702 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:09:30.823563    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-serving-cert: secret "maas-api-e2e-mt-9696d332-serving-cert" not found
Jul 28 20:09:30.823702 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:09:30.823664    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls podName:a363cb81-a24b-44c4-8ae1-a8a33cbb6320 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:31.323641196 +0000 UTC m=+1989.784399781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls") pod "maas-api-e2e-mt-9696d332-8544c8c589-2jt9g" (UID: "a363cb81-a24b-44c4-8ae1-a8a33cbb6320") : secret "maas-api-e2e-mt-9696d332-serving-cert" not found
Jul 28 20:09:30.824303 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.824278    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-openshift-service-ca\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:30.840910 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:30.833544    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xm5\" (UniqueName: \"kubernetes.io/projected/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-kube-api-access-m7xm5\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:31.327661 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:31.327627    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:31.327854 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:09:31.327761    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-serving-cert: secret "maas-api-e2e-mt-9696d332-serving-cert" not found
Jul 28 20:09:31.327854 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:09:31.327828    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls podName:a363cb81-a24b-44c4-8ae1-a8a33cbb6320 nodeName:}" failed. No retries permitted until 2026-07-28 20:09:32.327808495 +0000 UTC m=+1990.788567074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls") pod "maas-api-e2e-mt-9696d332-8544c8c589-2jt9g" (UID: "a363cb81-a24b-44c4-8ae1-a8a33cbb6320") : secret "maas-api-e2e-mt-9696d332-serving-cert" not found
Jul 28 20:09:31.377342 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:31.377265    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:31.493578 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:31.493543    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:09:32.334764 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:32.334722    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:32.337109 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:32.337085    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls\") pod \"maas-api-e2e-mt-9696d332-8544c8c589-2jt9g\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:32.442754 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:32.442728    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:32.567013 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:32.566987    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"]
Jul 28 20:09:32.569782 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:09:32.569757    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda363cb81_a24b_44c4_8ae1_a8a33cbb6320.slice/crio-c495b6fe30f949c00efbccb37ee50c7e69ea6be23a64ed4062f8508541307ab0 WatchSource:0}: Error finding container c495b6fe30f949c00efbccb37ee50c7e69ea6be23a64ed4062f8508541307ab0: Status 404 returned error can't find the container with id c495b6fe30f949c00efbccb37ee50c7e69ea6be23a64ed4062f8508541307ab0
Jul 28 20:09:33.276673 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:33.276599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g" event={"ID":"a363cb81-a24b-44c4-8ae1-a8a33cbb6320","Type":"ContainerStarted","Data":"06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d"}
Jul 28 20:09:33.276673 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:33.276653    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g" event={"ID":"a363cb81-a24b-44c4-8ae1-a8a33cbb6320","Type":"ContainerStarted","Data":"c495b6fe30f949c00efbccb37ee50c7e69ea6be23a64ed4062f8508541307ab0"}
Jul 28 20:09:33.276826 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:33.276714    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:09:33.292784 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:33.292731    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g" podStartSLOduration=2.8409014519999998 podStartE2EDuration="3.292718658s" totalImagesPullingTime="451.817206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:09:33.291168369 +0000 UTC m=+1991.751926968" watchObservedRunningTime="2026-07-28 20:09:33.292718658 +0000 UTC m=+1991.753477256"
Jul 28 20:09:36.299976 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:36.299941    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:09:39.286962 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:09:39.286935    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:10:00.487960 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:00.487918    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:01.209085 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:01.209052    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:01.310135 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:01.310105    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:01.661456 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:01.661425    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:07.357802 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:07.357766    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:07.422737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:07.422699    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:18.124296 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.124257    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"]
Jul 28 20:10:18.124813 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.124515    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g" podUID="a363cb81-a24b-44c4-8ae1-a8a33cbb6320" containerName="maas-api" containerID="cri-o://06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d" gracePeriod=30
Jul 28 20:10:18.154148 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.154122    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:18.259789 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.259760    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:18.329677 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.329650    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:18.358364 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.358343    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:10:18.403085 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.403028    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:18.474596 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.474570    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls\") pod \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") "
Jul 28 20:10:18.474717 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.474629    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7xm5\" (UniqueName: \"kubernetes.io/projected/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-kube-api-access-m7xm5\") pod \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") "
Jul 28 20:10:18.474717 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.474671    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-openshift-service-ca\") pod \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\" (UID: \"a363cb81-a24b-44c4-8ae1-a8a33cbb6320\") "
Jul 28 20:10:18.474717 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.474696    2555 generic.go:356] "Generic (PLEG): container finished" podID="a363cb81-a24b-44c4-8ae1-a8a33cbb6320" containerID="06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d" exitCode=0
Jul 28 20:10:18.474879 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.474745    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g" event={"ID":"a363cb81-a24b-44c4-8ae1-a8a33cbb6320","Type":"ContainerDied","Data":"06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d"}
Jul 28 20:10:18.474879 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.474792    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g" event={"ID":"a363cb81-a24b-44c4-8ae1-a8a33cbb6320","Type":"ContainerDied","Data":"c495b6fe30f949c00efbccb37ee50c7e69ea6be23a64ed4062f8508541307ab0"}
Jul 28 20:10:18.474879 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.474823    2555 scope.go:117] "RemoveContainer" containerID="06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d"
Jul 28 20:10:18.474879 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.474845    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"
Jul 28 20:10:18.475157 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.475099    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a363cb81-a24b-44c4-8ae1-a8a33cbb6320" (UID: "a363cb81-a24b-44c4-8ae1-a8a33cbb6320"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:10:18.476747 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.476721    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a363cb81-a24b-44c4-8ae1-a8a33cbb6320" (UID: "a363cb81-a24b-44c4-8ae1-a8a33cbb6320"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:10:18.476939 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.476914    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-kube-api-access-m7xm5" (OuterVolumeSpecName: "kube-api-access-m7xm5") pod "a363cb81-a24b-44c4-8ae1-a8a33cbb6320" (UID: "a363cb81-a24b-44c4-8ae1-a8a33cbb6320"). InnerVolumeSpecName "kube-api-access-m7xm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:10:18.491056 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.491036    2555 scope.go:117] "RemoveContainer" containerID="06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d"
Jul 28 20:10:18.491297 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:10:18.491276    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d\": container with ID starting with 06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d not found: ID does not exist" containerID="06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d"
Jul 28 20:10:18.491362 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.491308    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d"} err="failed to get container status \"06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d\": rpc error: code = NotFound desc = could not find container \"06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d\": container with ID starting with 06bc4ee5dfb56c50105273c5a8824a0c7737a7d086f3b508f6155bb3692a200d not found: ID does not exist"
Jul 28 20:10:18.575783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.575757    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:10:18.575783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.575781    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7xm5\" (UniqueName: \"kubernetes.io/projected/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-kube-api-access-m7xm5\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:10:18.575928 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.575795    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a363cb81-a24b-44c4-8ae1-a8a33cbb6320-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:10:18.684762 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.684710    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:18.800357 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.800331    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"]
Jul 28 20:10:18.804136 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:18.804106    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9696d332-8544c8c589-2jt9g"]
Jul 28 20:10:19.984931 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:19.984866    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a363cb81-a24b-44c4-8ae1-a8a33cbb6320" path="/var/lib/kubelet/pods/a363cb81-a24b-44c4-8ae1-a8a33cbb6320/volumes"
Jul 28 20:10:21.102846 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:21.102807    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:10:24.494149 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:24.494116    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:24.558832 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:24.558799    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:24.745451 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:24.745368    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:24.963585 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:24.963551    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:25.097077 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:25.097049    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:25.195897 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:25.195869    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:25.318648 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:25.318621    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:28.762211 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:28.762172    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:31.290011 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:31.289974    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:31.403295 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:31.403260    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:31.490125 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:31.490096    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:31.647141 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:31.647110    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:37.331494 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:37.331456    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:37.461882 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:37.461854    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:44.265711 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:44.265615    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:44.597081 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:44.597054    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:44.871638 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:44.871563    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:47.778109 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:47.778074    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:47.864212 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:47.864179    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:48.021016 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.020984    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"]
Jul 28 20:10:48.021285 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.021260    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a363cb81-a24b-44c4-8ae1-a8a33cbb6320" containerName="maas-api"
Jul 28 20:10:48.021285 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.021276    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a363cb81-a24b-44c4-8ae1-a8a33cbb6320" containerName="maas-api"
Jul 28 20:10:48.021462 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.021337    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a363cb81-a24b-44c4-8ae1-a8a33cbb6320" containerName="maas-api"
Jul 28 20:10:48.024281 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.024262    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.026867 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.026836    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xsbnd\""
Jul 28 20:10:48.027014 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.026991    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:10:48.027266 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.027249    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d517388f-serving-cert\""
Jul 28 20:10:48.027828 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.027806    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:10:48.037736 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.037709    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"]
Jul 28 20:10:48.057862 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.057837    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:48.086044 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.086020    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9phj\" (UniqueName: \"kubernetes.io/projected/5b182074-51c8-4c12-bc80-e3416e94a860-kube-api-access-x9phj\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.086150 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.086093    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b182074-51c8-4c12-bc80-e3416e94a860-openshift-service-ca\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.086205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.086146    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.187449 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.187426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b182074-51c8-4c12-bc80-e3416e94a860-openshift-service-ca\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.187570 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.187480    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.187570 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.187529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9phj\" (UniqueName: \"kubernetes.io/projected/5b182074-51c8-4c12-bc80-e3416e94a860-kube-api-access-x9phj\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.187692 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:10:48.187642    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-serving-cert: secret "maas-api-e2e-mt-d517388f-serving-cert" not found
Jul 28 20:10:48.187749 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:10:48.187724    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls podName:5b182074-51c8-4c12-bc80-e3416e94a860 nodeName:}" failed. No retries permitted until 2026-07-28 20:10:48.687703614 +0000 UTC m=+2067.148462202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls") pod "maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk" (UID: "5b182074-51c8-4c12-bc80-e3416e94a860") : secret "maas-api-e2e-mt-d517388f-serving-cert" not found
Jul 28 20:10:48.188123 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.188108    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b182074-51c8-4c12-bc80-e3416e94a860-openshift-service-ca\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.201532 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.201503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9phj\" (UniqueName: \"kubernetes.io/projected/5b182074-51c8-4c12-bc80-e3416e94a860-kube-api-access-x9phj\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.659578 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.659552    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:48.690130 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.690107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.692382 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.692354    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls\") pod \"maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.853931 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.853882    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:48.934752 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.934690    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:48.954227 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.954169    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:10:48.976335 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.976305    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"]
Jul 28 20:10:48.979635 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.979620    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:48.982965 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.982668    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d517388f\""
Jul 28 20:10:48.983075 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.982964    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d517388f-dockercfg-tcrvj\""
Jul 28 20:10:48.992054 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:48.992006    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"]
Jul 28 20:10:49.066393 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.066367    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"]
Jul 28 20:10:49.069372 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:10:49.069346    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b182074_51c8_4c12_bc80_e3416e94a860.slice/crio-b9a45a1e8201b0f936ac125d29df949ea7f90218f51711240e3509101f941cc3 WatchSource:0}: Error finding container b9a45a1e8201b0f936ac125d29df949ea7f90218f51711240e3509101f941cc3: Status 404 returned error can't find the container with id b9a45a1e8201b0f936ac125d29df949ea7f90218f51711240e3509101f941cc3
Jul 28 20:10:49.093937 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.093913    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhg74\" (UniqueName: \"kubernetes.io/projected/7085b861-3a38-4195-b9d5-87bbeb3133f9-kube-api-access-bhg74\") pod \"payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6\" (UID: \"7085b861-3a38-4195-b9d5-87bbeb3133f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:49.094040 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.094006    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7085b861-3a38-4195-b9d5-87bbeb3133f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6\" (UID: \"7085b861-3a38-4195-b9d5-87bbeb3133f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:49.195351 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.195289    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhg74\" (UniqueName: \"kubernetes.io/projected/7085b861-3a38-4195-b9d5-87bbeb3133f9-kube-api-access-bhg74\") pod \"payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6\" (UID: \"7085b861-3a38-4195-b9d5-87bbeb3133f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:49.195452 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.195357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7085b861-3a38-4195-b9d5-87bbeb3133f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6\" (UID: \"7085b861-3a38-4195-b9d5-87bbeb3133f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:49.196178 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.196159    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7085b861-3a38-4195-b9d5-87bbeb3133f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6\" (UID: \"7085b861-3a38-4195-b9d5-87bbeb3133f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:49.203782 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.203760    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhg74\" (UniqueName: \"kubernetes.io/projected/7085b861-3a38-4195-b9d5-87bbeb3133f9-kube-api-access-bhg74\") pod \"payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6\" (UID: \"7085b861-3a38-4195-b9d5-87bbeb3133f9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:49.294547 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.294521    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:49.415054 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.415025    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"]
Jul 28 20:10:49.418123 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:10:49.418093    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7085b861_3a38_4195_b9d5_87bbeb3133f9.slice/crio-c395cb6399443f72690dac302d6e748a2d2917ba9d6ddf140a9f536bec6a6b22 WatchSource:0}: Error finding container c395cb6399443f72690dac302d6e748a2d2917ba9d6ddf140a9f536bec6a6b22: Status 404 returned error can't find the container with id c395cb6399443f72690dac302d6e748a2d2917ba9d6ddf140a9f536bec6a6b22
Jul 28 20:10:49.609952 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.609920    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6" event={"ID":"7085b861-3a38-4195-b9d5-87bbeb3133f9","Type":"ContainerStarted","Data":"1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91"}
Jul 28 20:10:49.610090 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.609977    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6" event={"ID":"7085b861-3a38-4195-b9d5-87bbeb3133f9","Type":"ContainerStarted","Data":"c395cb6399443f72690dac302d6e748a2d2917ba9d6ddf140a9f536bec6a6b22"}
Jul 28 20:10:49.610090 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.610009    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:10:49.611391 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.611369    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk" event={"ID":"5b182074-51c8-4c12-bc80-e3416e94a860","Type":"ContainerStarted","Data":"a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038"}
Jul 28 20:10:49.611496 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.611411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk" event={"ID":"5b182074-51c8-4c12-bc80-e3416e94a860","Type":"ContainerStarted","Data":"b9a45a1e8201b0f936ac125d29df949ea7f90218f51711240e3509101f941cc3"}
Jul 28 20:10:49.611546 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.611497    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:10:49.627944 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.627875    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6" podStartSLOduration=1.627858134 podStartE2EDuration="1.627858134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:10:49.626521532 +0000 UTC m=+2068.087280131" watchObservedRunningTime="2026-07-28 20:10:49.627858134 +0000 UTC m=+2068.088616733"
Jul 28 20:10:49.646261 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:49.646216    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk" podStartSLOduration=2.27687789 podStartE2EDuration="2.646200421s" totalImagesPullingTime="369.322531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:10:49.645018245 +0000 UTC m=+2068.105776845" watchObservedRunningTime="2026-07-28 20:10:49.646200421 +0000 UTC m=+2068.106959010"
Jul 28 20:10:54.404790 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:54.404746    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:10:55.620814 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:10:55.620782    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:11:00.617858 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:00.617823    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:11:13.130187 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:13.130151    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:16.905783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.905743    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"]
Jul 28 20:11:16.909262 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.909236    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:16.913136 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.913113    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d517388f\"/\"openshift-service-ca.crt\""
Jul 28 20:11:16.914134 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.914112    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d517388f\"/\"kube-root-ca.crt\""
Jul 28 20:11:16.914247 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.914114    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d517388f\"/\"e2e-lifecycle-model-d517388f-kserve-self-signed-certs\""
Jul 28 20:11:16.914317 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.914242    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d517388f\"/\"default-dockercfg-5x4zp\""
Jul 28 20:11:16.919941 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.919917    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"]
Jul 28 20:11:16.927195 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.927173    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:11:16.982156 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.982132    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-dshm\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:16.982256 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.982166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-tmp-dir\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:16.982256 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.982207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5b6m\" (UniqueName: \"kubernetes.io/projected/732d8a03-0065-4863-a843-c66d1cc501d7-kube-api-access-k5b6m\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:16.982349 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.982253    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-home\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:16.982349 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.982280    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-model-cache\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:16.982349 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:16.982316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/732d8a03-0065-4863-a843-c66d1cc501d7-tls-certs\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083399 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083360    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-tmp-dir\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083510 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083447    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5b6m\" (UniqueName: \"kubernetes.io/projected/732d8a03-0065-4863-a843-c66d1cc501d7-kube-api-access-k5b6m\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083510 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083479    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-home\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083510 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083502    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-model-cache\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083675 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083608    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/732d8a03-0065-4863-a843-c66d1cc501d7-tls-certs\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083675 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-dshm\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083782 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-tmp-dir\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083838 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-home\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.083950 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.083931    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-model-cache\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.085756 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.085737    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-dshm\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.086195 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.086173    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/732d8a03-0065-4863-a843-c66d1cc501d7-tls-certs\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.097420 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.097394    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5b6m\" (UniqueName: \"kubernetes.io/projected/732d8a03-0065-4863-a843-c66d1cc501d7-kube-api-access-k5b6m\") pod \"e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") " pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.219473 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.219418    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:17.349174 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.349151    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"]
Jul 28 20:11:17.351284 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:11:17.351256    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod732d8a03_0065_4863_a843_c66d1cc501d7.slice/crio-42a62e8ca54e538c04ff2e40dc3a18f61462e15ccbcbe157bad65987ab863094 WatchSource:0}: Error finding container 42a62e8ca54e538c04ff2e40dc3a18f61462e15ccbcbe157bad65987ab863094: Status 404 returned error can't find the container with id 42a62e8ca54e538c04ff2e40dc3a18f61462e15ccbcbe157bad65987ab863094
Jul 28 20:11:17.736466 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.736432    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s" event={"ID":"732d8a03-0065-4863-a843-c66d1cc501d7","Type":"ContainerStarted","Data":"fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b"}
Jul 28 20:11:17.736660 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.736478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s" event={"ID":"732d8a03-0065-4863-a843-c66d1cc501d7","Type":"ContainerStarted","Data":"42a62e8ca54e538c04ff2e40dc3a18f61462e15ccbcbe157bad65987ab863094"}
Jul 28 20:11:17.758818 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:17.758758    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s" podStartSLOduration=1.758742858 podStartE2EDuration="1.758742858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:11:17.757490756 +0000 UTC m=+2096.218249354" watchObservedRunningTime="2026-07-28 20:11:17.758742858 +0000 UTC m=+2096.219501470"
Jul 28 20:11:20.527353 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:20.527315    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:11:27.220598 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:27.220553    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:27.221094 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:27.220616    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:27.233070 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:27.233044    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:11:27.792842 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:11:27.792815    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:12:00.809983 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:00.809945    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:12:06.808265 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:06.808224    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:22.705107 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:22.705035    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:12:23.973677 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:23.973640    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"]
Jul 28 20:12:23.974126 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:23.973848    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk" podUID="5b182074-51c8-4c12-bc80-e3416e94a860" containerName="maas-api" containerID="cri-o://a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038" gracePeriod=30
Jul 28 20:12:24.016657 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.016631    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:24.049044 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.049021    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"]
Jul 28 20:12:24.049250 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.049223    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6" podUID="7085b861-3a38-4195-b9d5-87bbeb3133f9" containerName="payload-pre-processing" containerID="cri-o://1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91" gracePeriod=30
Jul 28 20:12:24.117371 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.117343    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:24.216540 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.216514    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:24.321695 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.321672    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:12:24.324555 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.324535    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:12:24.455947 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.455921    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7085b861-3a38-4195-b9d5-87bbeb3133f9-plugins-config-volume\") pod \"7085b861-3a38-4195-b9d5-87bbeb3133f9\" (UID: \"7085b861-3a38-4195-b9d5-87bbeb3133f9\") "
Jul 28 20:12:24.456100 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.455957    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b182074-51c8-4c12-bc80-e3416e94a860-openshift-service-ca\") pod \"5b182074-51c8-4c12-bc80-e3416e94a860\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") "
Jul 28 20:12:24.456100 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.455973    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls\") pod \"5b182074-51c8-4c12-bc80-e3416e94a860\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") "
Jul 28 20:12:24.456100 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.456030    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9phj\" (UniqueName: \"kubernetes.io/projected/5b182074-51c8-4c12-bc80-e3416e94a860-kube-api-access-x9phj\") pod \"5b182074-51c8-4c12-bc80-e3416e94a860\" (UID: \"5b182074-51c8-4c12-bc80-e3416e94a860\") "
Jul 28 20:12:24.456100 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.456069    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhg74\" (UniqueName: \"kubernetes.io/projected/7085b861-3a38-4195-b9d5-87bbeb3133f9-kube-api-access-bhg74\") pod \"7085b861-3a38-4195-b9d5-87bbeb3133f9\" (UID: \"7085b861-3a38-4195-b9d5-87bbeb3133f9\") "
Jul 28 20:12:24.456322 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.456293    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7085b861-3a38-4195-b9d5-87bbeb3133f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7085b861-3a38-4195-b9d5-87bbeb3133f9" (UID: "7085b861-3a38-4195-b9d5-87bbeb3133f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:12:24.456384 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.456318    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b182074-51c8-4c12-bc80-e3416e94a860-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b182074-51c8-4c12-bc80-e3416e94a860" (UID: "5b182074-51c8-4c12-bc80-e3416e94a860"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:12:24.458188 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.458165    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7085b861-3a38-4195-b9d5-87bbeb3133f9-kube-api-access-bhg74" (OuterVolumeSpecName: "kube-api-access-bhg74") pod "7085b861-3a38-4195-b9d5-87bbeb3133f9" (UID: "7085b861-3a38-4195-b9d5-87bbeb3133f9"). InnerVolumeSpecName "kube-api-access-bhg74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:12:24.458188 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.458179    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b182074-51c8-4c12-bc80-e3416e94a860-kube-api-access-x9phj" (OuterVolumeSpecName: "kube-api-access-x9phj") pod "5b182074-51c8-4c12-bc80-e3416e94a860" (UID: "5b182074-51c8-4c12-bc80-e3416e94a860"). InnerVolumeSpecName "kube-api-access-x9phj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:12:24.458328 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.458262    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b182074-51c8-4c12-bc80-e3416e94a860" (UID: "5b182074-51c8-4c12-bc80-e3416e94a860"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:12:24.556729 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.556708    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7085b861-3a38-4195-b9d5-87bbeb3133f9-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:24.556729 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.556728    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b182074-51c8-4c12-bc80-e3416e94a860-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:24.556841 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.556740    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b182074-51c8-4c12-bc80-e3416e94a860-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:24.556841 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.556748    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9phj\" (UniqueName: \"kubernetes.io/projected/5b182074-51c8-4c12-bc80-e3416e94a860-kube-api-access-x9phj\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:24.556841 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.556757    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhg74\" (UniqueName: \"kubernetes.io/projected/7085b861-3a38-4195-b9d5-87bbeb3133f9-kube-api-access-bhg74\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:24.727287 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:24.727254    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:25.042521 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.042491    2555 generic.go:356] "Generic (PLEG): container finished" podID="5b182074-51c8-4c12-bc80-e3416e94a860" containerID="a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038" exitCode=0
Jul 28 20:12:25.042925 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.042533    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"
Jul 28 20:12:25.042925 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.042578    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk" event={"ID":"5b182074-51c8-4c12-bc80-e3416e94a860","Type":"ContainerDied","Data":"a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038"}
Jul 28 20:12:25.042925 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.042629    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk" event={"ID":"5b182074-51c8-4c12-bc80-e3416e94a860","Type":"ContainerDied","Data":"b9a45a1e8201b0f936ac125d29df949ea7f90218f51711240e3509101f941cc3"}
Jul 28 20:12:25.042925 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.042646    2555 scope.go:117] "RemoveContainer" containerID="a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038"
Jul 28 20:12:25.044369 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.044339    2555 generic.go:356] "Generic (PLEG): container finished" podID="7085b861-3a38-4195-b9d5-87bbeb3133f9" containerID="1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91" exitCode=0
Jul 28 20:12:25.044468 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.044393    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"
Jul 28 20:12:25.044468 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.044420    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6" event={"ID":"7085b861-3a38-4195-b9d5-87bbeb3133f9","Type":"ContainerDied","Data":"1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91"}
Jul 28 20:12:25.044548 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.044472    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6" event={"ID":"7085b861-3a38-4195-b9d5-87bbeb3133f9","Type":"ContainerDied","Data":"c395cb6399443f72690dac302d6e748a2d2917ba9d6ddf140a9f536bec6a6b22"}
Jul 28 20:12:25.053146 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.053096    2555 scope.go:117] "RemoveContainer" containerID="a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038"
Jul 28 20:12:25.053470 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:12:25.053446    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038\": container with ID starting with a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038 not found: ID does not exist" containerID="a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038"
Jul 28 20:12:25.053583 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.053474    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038"} err="failed to get container status \"a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038\": rpc error: code = NotFound desc = could not find container \"a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038\": container with ID starting with a1d161cf7f69d802960c1a3a80551f4b547bb1da9f87c103d8ad8fcdfffa1038 not found: ID does not exist"
Jul 28 20:12:25.053583 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.053492    2555 scope.go:117] "RemoveContainer" containerID="1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91"
Jul 28 20:12:25.061263 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.061241    2555 scope.go:117] "RemoveContainer" containerID="1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91"
Jul 28 20:12:25.061552 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:12:25.061532    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91\": container with ID starting with 1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91 not found: ID does not exist" containerID="1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91"
Jul 28 20:12:25.061615 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.061560    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91"} err="failed to get container status \"1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91\": rpc error: code = NotFound desc = could not find container \"1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91\": container with ID starting with 1faef01a21af419f8aae4846aa327d8f3f32b38c0109ded18df70179287d9c91 not found: ID does not exist"
Jul 28 20:12:25.067028 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.067001    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"]
Jul 28 20:12:25.072827 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.072801    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d517388f-96cb7b4c9-pt4fk"]
Jul 28 20:12:25.084820 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.084794    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"]
Jul 28 20:12:25.090113 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.090093    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d517388f-74955c64d-dfvd6"]
Jul 28 20:12:25.988022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.987974    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b182074-51c8-4c12-bc80-e3416e94a860" path="/var/lib/kubelet/pods/5b182074-51c8-4c12-bc80-e3416e94a860/volumes"
Jul 28 20:12:25.988466 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:25.988439    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7085b861-3a38-4195-b9d5-87bbeb3133f9" path="/var/lib/kubelet/pods/7085b861-3a38-4195-b9d5-87bbeb3133f9/volumes"
Jul 28 20:12:27.398387 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:27.398349    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:32.376482 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:32.376444    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"]
Jul 28 20:12:32.472108 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:32.472068    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:12:33.600990 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:33.600951    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:12:47.395943 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.395871    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s" podUID="732d8a03-0065-4863-a843-c66d1cc501d7" containerName="main" containerID="cri-o://fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b" gracePeriod=45
Jul 28 20:12:47.639020 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.638995    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:12:47.730474 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730403    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-tmp-dir\") pod \"732d8a03-0065-4863-a843-c66d1cc501d7\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") "
Jul 28 20:12:47.730474 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730440    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5b6m\" (UniqueName: \"kubernetes.io/projected/732d8a03-0065-4863-a843-c66d1cc501d7-kube-api-access-k5b6m\") pod \"732d8a03-0065-4863-a843-c66d1cc501d7\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") "
Jul 28 20:12:47.730663 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730507    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-home\") pod \"732d8a03-0065-4863-a843-c66d1cc501d7\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") "
Jul 28 20:12:47.730663 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730533    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-model-cache\") pod \"732d8a03-0065-4863-a843-c66d1cc501d7\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") "
Jul 28 20:12:47.730663 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730567    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/732d8a03-0065-4863-a843-c66d1cc501d7-tls-certs\") pod \"732d8a03-0065-4863-a843-c66d1cc501d7\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") "
Jul 28 20:12:47.730663 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730613    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-dshm\") pod \"732d8a03-0065-4863-a843-c66d1cc501d7\" (UID: \"732d8a03-0065-4863-a843-c66d1cc501d7\") "
Jul 28 20:12:47.730836 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730658    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "732d8a03-0065-4863-a843-c66d1cc501d7" (UID: "732d8a03-0065-4863-a843-c66d1cc501d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:12:47.730836 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730765    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-home" (OuterVolumeSpecName: "home") pod "732d8a03-0065-4863-a843-c66d1cc501d7" (UID: "732d8a03-0065-4863-a843-c66d1cc501d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:12:47.730836 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730778    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "732d8a03-0065-4863-a843-c66d1cc501d7" (UID: "732d8a03-0065-4863-a843-c66d1cc501d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:12:47.731006 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730858    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:47.731006 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730908    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:47.731006 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.730922    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:47.732695 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.732668    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/732d8a03-0065-4863-a843-c66d1cc501d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "732d8a03-0065-4863-a843-c66d1cc501d7" (UID: "732d8a03-0065-4863-a843-c66d1cc501d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:12:47.732799 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.732709    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-dshm" (OuterVolumeSpecName: "dshm") pod "732d8a03-0065-4863-a843-c66d1cc501d7" (UID: "732d8a03-0065-4863-a843-c66d1cc501d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:12:47.732799 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.732729    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732d8a03-0065-4863-a843-c66d1cc501d7-kube-api-access-k5b6m" (OuterVolumeSpecName: "kube-api-access-k5b6m") pod "732d8a03-0065-4863-a843-c66d1cc501d7" (UID: "732d8a03-0065-4863-a843-c66d1cc501d7"). InnerVolumeSpecName "kube-api-access-k5b6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:12:47.831698 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.831674    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/732d8a03-0065-4863-a843-c66d1cc501d7-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:47.831698 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.831696    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5b6m\" (UniqueName: \"kubernetes.io/projected/732d8a03-0065-4863-a843-c66d1cc501d7-kube-api-access-k5b6m\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:47.831834 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:47.831707    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/732d8a03-0065-4863-a843-c66d1cc501d7-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:12:48.146117 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.146089    2555 generic.go:356] "Generic (PLEG): container finished" podID="732d8a03-0065-4863-a843-c66d1cc501d7" containerID="fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b" exitCode=0
Jul 28 20:12:48.146243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.146129    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"
Jul 28 20:12:48.146243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.146140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s" event={"ID":"732d8a03-0065-4863-a843-c66d1cc501d7","Type":"ContainerDied","Data":"fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b"}
Jul 28 20:12:48.146243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.146190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s" event={"ID":"732d8a03-0065-4863-a843-c66d1cc501d7","Type":"ContainerDied","Data":"42a62e8ca54e538c04ff2e40dc3a18f61462e15ccbcbe157bad65987ab863094"}
Jul 28 20:12:48.146243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.146214    2555 scope.go:117] "RemoveContainer" containerID="fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b"
Jul 28 20:12:48.154362 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.154336    2555 scope.go:117] "RemoveContainer" containerID="fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b"
Jul 28 20:12:48.154622 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:12:48.154601    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b\": container with ID starting with fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b not found: ID does not exist" containerID="fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b"
Jul 28 20:12:48.154669 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.154630    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b"} err="failed to get container status \"fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b\": rpc error: code = NotFound desc = could not find container \"fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b\": container with ID starting with fd2496b9afa73b5a59502dddd2b1a8fa20706435b901bd569a29c6a45c763c9b not found: ID does not exist"
Jul 28 20:12:48.168528 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.168506    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"]
Jul 28 20:12:48.172897 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:48.172863    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d517388f/e2e-lifecycle-model-d517388f-kserve-9bbd65c55-6nn7s"]
Jul 28 20:12:49.983823 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:12:49.983780    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732d8a03-0065-4863-a843-c66d1cc501d7" path="/var/lib/kubelet/pods/732d8a03-0065-4863-a843-c66d1cc501d7/volumes"
Jul 28 20:13:09.811415 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:09.811380    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:10.442637 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:10.442596    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:17.485484 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:17.485445    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:17.634098 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:17.634061    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:17.820633 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:17.820594    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:18.042470 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:18.042439    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:19.209284 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:19.209255    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:19.311377 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:19.311350    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:20.300081 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:20.300046    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:20.397784 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:20.397758    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:21.539345 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:21.539312    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:13:21.840681 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:21.840650    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:28.093588 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:28.093557    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:28.195090 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:28.195060    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:28.632473 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:28.632443    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:31.321381 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:31.321346    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:13:35.050108 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:35.050074    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:35.163964 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:35.163933    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:35.364927 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:35.364830    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:45.768932 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:45.768833    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:13:46.119937 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:13:46.119878    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:07.088878 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:07.088839    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:13.409938 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:13.409904    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:13.510712 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:13.510681    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:13.743368 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:13.743291    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:14.136672 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:14.136641    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:14.212692 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:14.212666    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:16.872076 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.872041    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:16.970386 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970357    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"]
Jul 28 20:14:16.970762 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970731    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b182074-51c8-4c12-bc80-e3416e94a860" containerName="maas-api"
Jul 28 20:14:16.970762 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970750    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b182074-51c8-4c12-bc80-e3416e94a860" containerName="maas-api"
Jul 28 20:14:16.970762 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970767    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732d8a03-0065-4863-a843-c66d1cc501d7" containerName="main"
Jul 28 20:14:16.971023 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970775    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="732d8a03-0065-4863-a843-c66d1cc501d7" containerName="main"
Jul 28 20:14:16.971023 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970797    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7085b861-3a38-4195-b9d5-87bbeb3133f9" containerName="payload-pre-processing"
Jul 28 20:14:16.971023 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970804    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7085b861-3a38-4195-b9d5-87bbeb3133f9" containerName="payload-pre-processing"
Jul 28 20:14:16.971023 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970862    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b182074-51c8-4c12-bc80-e3416e94a860" containerName="maas-api"
Jul 28 20:14:16.971023 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970875    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7085b861-3a38-4195-b9d5-87bbeb3133f9" containerName="payload-pre-processing"
Jul 28 20:14:16.971023 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.970882    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="732d8a03-0065-4863-a843-c66d1cc501d7" containerName="main"
Jul 28 20:14:16.971224 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.971105    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:16.973879 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.973865    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:16.976642 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.976627    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-24876e-serving-cert\""
Jul 28 20:14:16.977477 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.977461    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:14:16.977784 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.977766    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:14:16.978863 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.978842    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xsbnd\""
Jul 28 20:14:16.993982 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:16.993963    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"]
Jul 28 20:14:17.080585 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.080544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e74d83-5953-4744-ba6f-92c2c7525bdf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.080713 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.080695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxp7p\" (UniqueName: \"kubernetes.io/projected/e0e74d83-5953-4744-ba6f-92c2c7525bdf-kube-api-access-bxp7p\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.080774 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.080739    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.082523 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.082495    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:17.181043 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.180980    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxp7p\" (UniqueName: \"kubernetes.io/projected/e0e74d83-5953-4744-ba6f-92c2c7525bdf-kube-api-access-bxp7p\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.181043 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.181029    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.181200 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.181075    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e74d83-5953-4744-ba6f-92c2c7525bdf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.181257 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:14:17.181201    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-serving-cert: secret "maas-api-e2e-inf-a-24876e-serving-cert" not found
Jul 28 20:14:17.181315 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:14:17.181277    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls podName:e0e74d83-5953-4744-ba6f-92c2c7525bdf nodeName:}" failed. No retries permitted until 2026-07-28 20:14:17.681258392 +0000 UTC m=+2276.142016983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls") pod "maas-api-e2e-inf-a-24876e-65597874d5-fh52t" (UID: "e0e74d83-5953-4744-ba6f-92c2c7525bdf") : secret "maas-api-e2e-inf-a-24876e-serving-cert" not found
Jul 28 20:14:17.181689 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.181669    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e74d83-5953-4744-ba6f-92c2c7525bdf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.194604 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.194576    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxp7p\" (UniqueName: \"kubernetes.io/projected/e0e74d83-5953-4744-ba6f-92c2c7525bdf-kube-api-access-bxp7p\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.324981 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.324951    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:17.587904 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.587862    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:17.683306 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.683282    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:17.685394 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.685365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.687644 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.687619    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls\") pod \"maas-api-e2e-inf-a-24876e-65597874d5-fh52t\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:17.772398 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.772376    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:17.884689 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:17.884632    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:18.015253 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:18.015222    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"]
Jul 28 20:14:18.018397 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:14:18.018365    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e74d83_5953_4744_ba6f_92c2c7525bdf.slice/crio-3a74e261f85c0a7ddc46804a9379af69b9a8ac3765e8d595eb9096898f406431 WatchSource:0}: Error finding container 3a74e261f85c0a7ddc46804a9379af69b9a8ac3765e8d595eb9096898f406431: Status 404 returned error can't find the container with id 3a74e261f85c0a7ddc46804a9379af69b9a8ac3765e8d595eb9096898f406431
Jul 28 20:14:18.019847 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:18.019831    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:14:18.548333 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:18.548305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t" event={"ID":"e0e74d83-5953-4744-ba6f-92c2c7525bdf","Type":"ContainerStarted","Data":"6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae"}
Jul 28 20:14:18.548448 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:18.548361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t" event={"ID":"e0e74d83-5953-4744-ba6f-92c2c7525bdf","Type":"ContainerStarted","Data":"3a74e261f85c0a7ddc46804a9379af69b9a8ac3765e8d595eb9096898f406431"}
Jul 28 20:14:18.548448 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:18.548440    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:18.569768 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:18.569723    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t" podStartSLOduration=2.133365794 podStartE2EDuration="2.569696426s" totalImagesPullingTime="436.330632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:18.569023485 +0000 UTC m=+2277.029782094" watchObservedRunningTime="2026-07-28 20:14:18.569696426 +0000 UTC m=+2277.030455025"
Jul 28 20:14:22.809782 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:22.809740    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:14:24.558560 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:24.558524    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:14:41.884830 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:41.884790    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:42.236077 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:42.235980    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:42.552989 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:42.552956    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:42.777664 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:42.777624    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:45.278324 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.278292    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:45.370106 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.370076    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"]
Jul 28 20:14:45.373746 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.373727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.380775 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.380756    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ab7491-serving-cert\""
Jul 28 20:14:45.397055 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.397033    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"]
Jul 28 20:14:45.426810 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.426788    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:45.481481 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.481458    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.481567 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.481497    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.481567 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.481528    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vfjw\" (UniqueName: \"kubernetes.io/projected/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-kube-api-access-2vfjw\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.550073 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.550051    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:45.582275 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.582257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.582359 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.582296    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.582359 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.582328    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vfjw\" (UniqueName: \"kubernetes.io/projected/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-kube-api-access-2vfjw\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.582433 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:14:45.582390    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-serving-cert: secret "maas-api-e2e-inf-b-ab7491-serving-cert" not found
Jul 28 20:14:45.582468 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:14:45.582459    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls podName:9cee0485-7fd1-4c4c-934a-15e55d3a3c92 nodeName:}" failed. No retries permitted until 2026-07-28 20:14:46.082441389 +0000 UTC m=+2304.543199966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls") pod "maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7" (UID: "9cee0485-7fd1-4c4c-934a-15e55d3a3c92") : secret "maas-api-e2e-inf-b-ab7491-serving-cert" not found
Jul 28 20:14:45.582921 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.582906    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.608437 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.608413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vfjw\" (UniqueName: \"kubernetes.io/projected/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-kube-api-access-2vfjw\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:45.749403 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.749374    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:45.868034 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.867978    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:45.962769 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:45.962746    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:46.086717 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.086695    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:46.089085 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.089067    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:46.207590 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.207529    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:46.283455 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.283429    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:46.307260 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.307239    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:14:46.360288 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.359957    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"]
Jul 28 20:14:46.363542 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.363518    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:46.366553 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.366519    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ab7491\""
Jul 28 20:14:46.366685 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.366590    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ab7491-dockercfg-t6zht\""
Jul 28 20:14:46.380480 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.380456    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"]
Jul 28 20:14:46.419189 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.419153    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"]
Jul 28 20:14:46.421847 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:14:46.421825    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cee0485_7fd1_4c4c_934a_15e55d3a3c92.slice/crio-e3c5381eda0c55e233926e9860a35c4d35f86fcef82dfc281ab48e09c5b4807c WatchSource:0}: Error finding container e3c5381eda0c55e233926e9860a35c4d35f86fcef82dfc281ab48e09c5b4807c: Status 404 returned error can't find the container with id e3c5381eda0c55e233926e9860a35c4d35f86fcef82dfc281ab48e09c5b4807c
Jul 28 20:14:46.490222 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.490165    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97\" (UID: \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:46.490222 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.490195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5gl\" (UniqueName: \"kubernetes.io/projected/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-kube-api-access-cd5gl\") pod \"payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97\" (UID: \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:46.591450 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.591426    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97\" (UID: \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:46.591549 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.591453    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5gl\" (UniqueName: \"kubernetes.io/projected/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-kube-api-access-cd5gl\") pod \"payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97\" (UID: \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:46.592165 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.592147    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97\" (UID: \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:46.600249 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.600222    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5gl\" (UniqueName: \"kubernetes.io/projected/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-kube-api-access-cd5gl\") pod \"payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97\" (UID: \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:46.668113 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.668088    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7" event={"ID":"9cee0485-7fd1-4c4c-934a-15e55d3a3c92","Type":"ContainerStarted","Data":"e3c5381eda0c55e233926e9860a35c4d35f86fcef82dfc281ab48e09c5b4807c"}
Jul 28 20:14:46.675276 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.675256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:46.798998 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:46.798968    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"]
Jul 28 20:14:46.801777 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:14:46.801752    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4fa414_e18a_489c_bdd6_3b70d13b81c7.slice/crio-69e83d6404c888e5ec6b591b1e735f9d16255136341f6a8cc3e5792a9f52b1c4 WatchSource:0}: Error finding container 69e83d6404c888e5ec6b591b1e735f9d16255136341f6a8cc3e5792a9f52b1c4: Status 404 returned error can't find the container with id 69e83d6404c888e5ec6b591b1e735f9d16255136341f6a8cc3e5792a9f52b1c4
Jul 28 20:14:47.673360 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:47.673329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97" event={"ID":"7d4fa414-e18a-489c-bdd6-3b70d13b81c7","Type":"ContainerStarted","Data":"59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a"}
Jul 28 20:14:47.673778 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:47.673380    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:14:47.673778 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:47.673404    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97" event={"ID":"7d4fa414-e18a-489c-bdd6-3b70d13b81c7","Type":"ContainerStarted","Data":"69e83d6404c888e5ec6b591b1e735f9d16255136341f6a8cc3e5792a9f52b1c4"}
Jul 28 20:14:47.674855 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:47.674817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7" event={"ID":"9cee0485-7fd1-4c4c-934a-15e55d3a3c92","Type":"ContainerStarted","Data":"fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639"}
Jul 28 20:14:47.674993 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:47.674872    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:47.693563 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:47.693516    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97" podStartSLOduration=1.693502433 podStartE2EDuration="1.693502433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:47.691181321 +0000 UTC m=+2306.151939921" watchObservedRunningTime="2026-07-28 20:14:47.693502433 +0000 UTC m=+2306.154261031"
Jul 28 20:14:47.711110 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:47.711063    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7" podStartSLOduration=2.274477491 podStartE2EDuration="2.711049013s" totalImagesPullingTime="436.571522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:47.708633664 +0000 UTC m=+2306.169392275" watchObservedRunningTime="2026-07-28 20:14:47.711049013 +0000 UTC m=+2306.171807589"
Jul 28 20:14:52.303815 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:52.303778    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:14:53.685264 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:53.685228    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:14:58.681426 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:14:58.681388    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:15:10.233690 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:10.233623    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:17.404128 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.404045    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"]
Jul 28 20:15:17.407297 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.407277    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.410157 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.410133    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-24876e\"/\"default-dockercfg-fc5dr\""
Jul 28 20:15:17.410379 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.410364    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-24876e\"/\"openshift-service-ca.crt\""
Jul 28 20:15:17.411187 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.411169    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-24876e\"/\"kube-root-ca.crt\""
Jul 28 20:15:17.411249 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.411167    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-24876e\"/\"test-model-24876e-kserve-self-signed-certs\""
Jul 28 20:15:17.420074 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.420053    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"]
Jul 28 20:15:17.428762 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.428741    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:17.517649 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.517615    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-dshm\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.517758 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.517670    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-home\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.517758 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.517689    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g455d\" (UniqueName: \"kubernetes.io/projected/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-kube-api-access-g455d\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.517758 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.517731    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tmp-dir\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.517758 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.517745    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tls-certs\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.517918 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.517760    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-model-cache\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.618480 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.618456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-home\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.618566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.618485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g455d\" (UniqueName: \"kubernetes.io/projected/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-kube-api-access-g455d\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.618566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.618513    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tmp-dir\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.618566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.618527    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tls-certs\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.618566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.618543    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-model-cache\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.618772 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.618584    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-dshm\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.618928 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.618907    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-home\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.619005 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.618928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tmp-dir\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.619084 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.619058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-model-cache\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.620999 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.620983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-dshm\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.621190 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.621172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tls-certs\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.628237 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.628218    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g455d\" (UniqueName: \"kubernetes.io/projected/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-kube-api-access-g455d\") pod \"test-model-24876e-kserve-88c9db685-jrs86\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") " pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.717557 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.717498    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:17.840443 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:17.840421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"]
Jul 28 20:15:17.845602 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:15:17.845576    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ee0e0f8_35ad_4a2d_897f_be057fb90c54.slice/crio-19105cceccf1240d54f15aae6e38a7ce93ec230af048629a3daff3db0e7e7a2c WatchSource:0}: Error finding container 19105cceccf1240d54f15aae6e38a7ce93ec230af048629a3daff3db0e7e7a2c: Status 404 returned error can't find the container with id 19105cceccf1240d54f15aae6e38a7ce93ec230af048629a3daff3db0e7e7a2c
Jul 28 20:15:18.811934 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:18.811866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86" event={"ID":"9ee0e0f8-35ad-4a2d-897f-be057fb90c54","Type":"ContainerStarted","Data":"06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d"}
Jul 28 20:15:18.811934 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:18.811938    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86" event={"ID":"9ee0e0f8-35ad-4a2d-897f-be057fb90c54","Type":"ContainerStarted","Data":"19105cceccf1240d54f15aae6e38a7ce93ec230af048629a3daff3db0e7e7a2c"}
Jul 28 20:15:18.831882 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:18.831826    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86" podStartSLOduration=1.831811832 podStartE2EDuration="1.831811832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:18.830098512 +0000 UTC m=+2337.290857111" watchObservedRunningTime="2026-07-28 20:15:18.831811832 +0000 UTC m=+2337.292570432"
Jul 28 20:15:21.002683 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:21.002645    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:15:27.718661 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:27.718611    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:27.719171 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:27.718771    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:27.731945 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:27.731919    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:27.862855 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:27.862826    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:15:32.396503 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:32.396460    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:15:43.608101 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.608065    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"]
Jul 28 20:15:43.612009 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.611987    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.615735 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.615699    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ab7491\"/\"test-model-ab7491-kserve-self-signed-certs\""
Jul 28 20:15:43.615858 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.615700    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ab7491\"/\"default-dockercfg-dx5wg\""
Jul 28 20:15:43.615858 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.615756    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ab7491\"/\"openshift-service-ca.crt\""
Jul 28 20:15:43.615858 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.615702    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ab7491\"/\"kube-root-ca.crt\""
Jul 28 20:15:43.620343 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.620317    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"]
Jul 28 20:15:43.647896 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.647856    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:15:43.710107 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.710088    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tls-certs\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.710205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.710125    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-dshm\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.710205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.710153    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tmp-dir\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.710205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.710189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-home\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.710314 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.710213    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-model-cache\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.710314 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.710266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5c95\" (UniqueName: \"kubernetes.io/projected/e4151c4f-b915-413a-8fb3-a1ae92cfa081-kube-api-access-b5c95\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.810716 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.810692    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5c95\" (UniqueName: \"kubernetes.io/projected/e4151c4f-b915-413a-8fb3-a1ae92cfa081-kube-api-access-b5c95\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.810811 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.810730    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tls-certs\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.810811 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.810752    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-dshm\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.811024 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.811002    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tmp-dir\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.811091 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.811046    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-home\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.811150 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.811088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-model-cache\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.811402 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.811380    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tmp-dir\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.811499 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.811413    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-model-cache\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.812414 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.811842    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-home\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.818539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.818515    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-dshm\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.819171 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.819148    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tls-certs\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.821102 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.821079    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5c95\" (UniqueName: \"kubernetes.io/projected/e4151c4f-b915-413a-8fb3-a1ae92cfa081-kube-api-access-b5c95\") pod \"test-model-ab7491-kserve-85d965cdcc-ljv9q\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") " pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:43.922187 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:43.922131    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:44.050410 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:44.050384    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"]
Jul 28 20:15:44.052093 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:15:44.052062    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4151c4f_b915_413a_8fb3_a1ae92cfa081.slice/crio-23dc67db78846ec12c65f12a6296abef42c716f4575196bbb19cd394f0b6ab2c WatchSource:0}: Error finding container 23dc67db78846ec12c65f12a6296abef42c716f4575196bbb19cd394f0b6ab2c: Status 404 returned error can't find the container with id 23dc67db78846ec12c65f12a6296abef42c716f4575196bbb19cd394f0b6ab2c
Jul 28 20:15:44.933508 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:44.933463    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q" event={"ID":"e4151c4f-b915-413a-8fb3-a1ae92cfa081","Type":"ContainerStarted","Data":"49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c"}
Jul 28 20:15:44.933508 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:44.933513    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q" event={"ID":"e4151c4f-b915-413a-8fb3-a1ae92cfa081","Type":"ContainerStarted","Data":"23dc67db78846ec12c65f12a6296abef42c716f4575196bbb19cd394f0b6ab2c"}
Jul 28 20:15:44.951799 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:44.951747    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q" podStartSLOduration=1.9517302239999998 podStartE2EDuration="1.951730224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:44.949953679 +0000 UTC m=+2363.410712278" watchObservedRunningTime="2026-07-28 20:15:44.951730224 +0000 UTC m=+2363.412488822"
Jul 28 20:15:47.701482 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:47.701443    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:15:53.923014 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:53.922981    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:53.923563 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:53.923029    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:53.935343 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:53.935319    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:15:53.988205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:15:53.988179    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:16:01.407671 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:01.407613    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:16:24.869789 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:24.869756    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"]
Jul 28 20:16:24.897944 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:24.897917    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:25.756458 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:25.756413    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"]
Jul 28 20:16:26.333183 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:26.333146    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:29.503584 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:29.503544    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:16:36.401089 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.401052    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"]
Jul 28 20:16:36.401616 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.401333    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t" podUID="e0e74d83-5953-4744-ba6f-92c2c7525bdf" containerName="maas-api" containerID="cri-o://6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae" gracePeriod=30
Jul 28 20:16:36.468775 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.468738    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:36.569476 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.569451    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:36.633551 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.633526    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:36.845321 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.845301    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:16:36.908874 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.908849    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e74d83-5953-4744-ba6f-92c2c7525bdf-openshift-service-ca\") pod \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") "
Jul 28 20:16:36.908997 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.908930    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxp7p\" (UniqueName: \"kubernetes.io/projected/e0e74d83-5953-4744-ba6f-92c2c7525bdf-kube-api-access-bxp7p\") pod \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") "
Jul 28 20:16:36.908997 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.908949    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls\") pod \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\" (UID: \"e0e74d83-5953-4744-ba6f-92c2c7525bdf\") "
Jul 28 20:16:36.909265 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.909240    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e74d83-5953-4744-ba6f-92c2c7525bdf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0e74d83-5953-4744-ba6f-92c2c7525bdf" (UID: "e0e74d83-5953-4744-ba6f-92c2c7525bdf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:16:36.911007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.910977    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0e74d83-5953-4744-ba6f-92c2c7525bdf" (UID: "e0e74d83-5953-4744-ba6f-92c2c7525bdf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:16:36.911007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:36.910987    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e74d83-5953-4744-ba6f-92c2c7525bdf-kube-api-access-bxp7p" (OuterVolumeSpecName: "kube-api-access-bxp7p") pod "e0e74d83-5953-4744-ba6f-92c2c7525bdf" (UID: "e0e74d83-5953-4744-ba6f-92c2c7525bdf"). InnerVolumeSpecName "kube-api-access-bxp7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:37.004670 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.004603    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:37.009792 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.009776    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e74d83-5953-4744-ba6f-92c2c7525bdf-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:37.009844 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.009794    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxp7p\" (UniqueName: \"kubernetes.io/projected/e0e74d83-5953-4744-ba6f-92c2c7525bdf-kube-api-access-bxp7p\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:37.009844 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.009804    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0e74d83-5953-4744-ba6f-92c2c7525bdf-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:37.170576 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.170547    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0e74d83-5953-4744-ba6f-92c2c7525bdf" containerID="6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae" exitCode=0
Jul 28 20:16:37.170696 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.170592    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t" event={"ID":"e0e74d83-5953-4744-ba6f-92c2c7525bdf","Type":"ContainerDied","Data":"6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae"}
Jul 28 20:16:37.170696 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.170594    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"
Jul 28 20:16:37.170696 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.170635    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t" event={"ID":"e0e74d83-5953-4744-ba6f-92c2c7525bdf","Type":"ContainerDied","Data":"3a74e261f85c0a7ddc46804a9379af69b9a8ac3765e8d595eb9096898f406431"}
Jul 28 20:16:37.170696 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.170658    2555 scope.go:117] "RemoveContainer" containerID="6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae"
Jul 28 20:16:37.179029 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.179010    2555 scope.go:117] "RemoveContainer" containerID="6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae"
Jul 28 20:16:37.179339 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:16:37.179316    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae\": container with ID starting with 6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae not found: ID does not exist" containerID="6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae"
Jul 28 20:16:37.179401 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.179351    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae"} err="failed to get container status \"6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae\": rpc error: code = NotFound desc = could not find container \"6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae\": container with ID starting with 6bf94517e4f0e3f3e3350f2cb96a1961495746fa711c5c5366c90b24511fd7ae not found: ID does not exist"
Jul 28 20:16:37.191425 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.191403    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"]
Jul 28 20:16:37.195817 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.195797    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24876e-65597874d5-fh52t"]
Jul 28 20:16:37.984027 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:37.983960    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0e74d83-5953-4744-ba6f-92c2c7525bdf" path="/var/lib/kubelet/pods/e0e74d83-5953-4744-ba6f-92c2c7525bdf/volumes"
Jul 28 20:16:39.888620 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:39.888575    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86" podUID="9ee0e0f8-35ad-4a2d-897f-be057fb90c54" containerName="main" containerID="cri-o://06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d" gracePeriod=45
Jul 28 20:16:40.135450 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.135417    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:16:40.193861 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.193783    2555 generic.go:356] "Generic (PLEG): container finished" podID="9ee0e0f8-35ad-4a2d-897f-be057fb90c54" containerID="06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d" exitCode=0
Jul 28 20:16:40.193861 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.193833    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"
Jul 28 20:16:40.193861 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.193853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86" event={"ID":"9ee0e0f8-35ad-4a2d-897f-be057fb90c54","Type":"ContainerDied","Data":"06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d"}
Jul 28 20:16:40.194132 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.193908    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86" event={"ID":"9ee0e0f8-35ad-4a2d-897f-be057fb90c54","Type":"ContainerDied","Data":"19105cceccf1240d54f15aae6e38a7ce93ec230af048629a3daff3db0e7e7a2c"}
Jul 28 20:16:40.194132 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.193933    2555 scope.go:117] "RemoveContainer" containerID="06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d"
Jul 28 20:16:40.201432 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.201416    2555 scope.go:117] "RemoveContainer" containerID="06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d"
Jul 28 20:16:40.201685 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:16:40.201664    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d\": container with ID starting with 06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d not found: ID does not exist" containerID="06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d"
Jul 28 20:16:40.201749 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.201691    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d"} err="failed to get container status \"06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d\": rpc error: code = NotFound desc = could not find container \"06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d\": container with ID starting with 06e3cc2270156b263ca541c77caf2135a32a8436ca8830aab3ec366604ec285d not found: ID does not exist"
Jul 28 20:16:40.232379 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232359    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g455d\" (UniqueName: \"kubernetes.io/projected/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-kube-api-access-g455d\") pod \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") "
Jul 28 20:16:40.232462 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232398    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-dshm\") pod \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") "
Jul 28 20:16:40.232462 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232444    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-model-cache\") pod \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") "
Jul 28 20:16:40.232571 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232483    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-home\") pod \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") "
Jul 28 20:16:40.232571 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232523    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tls-certs\") pod \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") "
Jul 28 20:16:40.232681 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232583    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tmp-dir\") pod \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\" (UID: \"9ee0e0f8-35ad-4a2d-897f-be057fb90c54\") "
Jul 28 20:16:40.232740 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232723    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-home" (OuterVolumeSpecName: "home") pod "9ee0e0f8-35ad-4a2d-897f-be057fb90c54" (UID: "9ee0e0f8-35ad-4a2d-897f-be057fb90c54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:40.232740 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232730    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ee0e0f8-35ad-4a2d-897f-be057fb90c54" (UID: "9ee0e0f8-35ad-4a2d-897f-be057fb90c54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:40.232836 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232798    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:40.232836 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232814    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:40.233008 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.232985    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ee0e0f8-35ad-4a2d-897f-be057fb90c54" (UID: "9ee0e0f8-35ad-4a2d-897f-be057fb90c54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:40.234782 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.234757    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ee0e0f8-35ad-4a2d-897f-be057fb90c54" (UID: "9ee0e0f8-35ad-4a2d-897f-be057fb90c54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:16:40.234874 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.234802    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-dshm" (OuterVolumeSpecName: "dshm") pod "9ee0e0f8-35ad-4a2d-897f-be057fb90c54" (UID: "9ee0e0f8-35ad-4a2d-897f-be057fb90c54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:40.234874 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.234849    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-kube-api-access-g455d" (OuterVolumeSpecName: "kube-api-access-g455d") pod "9ee0e0f8-35ad-4a2d-897f-be057fb90c54" (UID: "9ee0e0f8-35ad-4a2d-897f-be057fb90c54"). InnerVolumeSpecName "kube-api-access-g455d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:40.333868 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.333846    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:40.333868 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.333868    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g455d\" (UniqueName: \"kubernetes.io/projected/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-kube-api-access-g455d\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:40.334004 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.333878    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:40.334004 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.333903    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ee0e0f8-35ad-4a2d-897f-be057fb90c54-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:40.517494 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.517469    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"]
Jul 28 20:16:40.521635 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.521597    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-24876e/test-model-24876e-kserve-88c9db685-jrs86"]
Jul 28 20:16:40.776064 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:40.775997    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q" podUID="e4151c4f-b915-413a-8fb3-a1ae92cfa081" containerName="main" containerID="cri-o://49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c" gracePeriod=45
Jul 28 20:16:41.022231 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.022205    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:16:41.038240 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038192    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tmp-dir\") pod \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") "
Jul 28 20:16:41.038331 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038244    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-dshm\") pod \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") "
Jul 28 20:16:41.038331 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038263    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-model-cache\") pod \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") "
Jul 28 20:16:41.038331 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038293    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tls-certs\") pod \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") "
Jul 28 20:16:41.038331 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038319    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-home\") pod \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") "
Jul 28 20:16:41.038545 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038337    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5c95\" (UniqueName: \"kubernetes.io/projected/e4151c4f-b915-413a-8fb3-a1ae92cfa081-kube-api-access-b5c95\") pod \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\" (UID: \"e4151c4f-b915-413a-8fb3-a1ae92cfa081\") "
Jul 28 20:16:41.038545 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038453    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4151c4f-b915-413a-8fb3-a1ae92cfa081" (UID: "e4151c4f-b915-413a-8fb3-a1ae92cfa081"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:41.038645 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038623    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4151c4f-b915-413a-8fb3-a1ae92cfa081" (UID: "e4151c4f-b915-413a-8fb3-a1ae92cfa081"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:41.038711 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038628    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:41.038785 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.038762    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-home" (OuterVolumeSpecName: "home") pod "e4151c4f-b915-413a-8fb3-a1ae92cfa081" (UID: "e4151c4f-b915-413a-8fb3-a1ae92cfa081"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:41.040364 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.040343    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-dshm" (OuterVolumeSpecName: "dshm") pod "e4151c4f-b915-413a-8fb3-a1ae92cfa081" (UID: "e4151c4f-b915-413a-8fb3-a1ae92cfa081"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:41.040471 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.040419    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4151c4f-b915-413a-8fb3-a1ae92cfa081" (UID: "e4151c4f-b915-413a-8fb3-a1ae92cfa081"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:16:41.040517 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.040479    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4151c4f-b915-413a-8fb3-a1ae92cfa081-kube-api-access-b5c95" (OuterVolumeSpecName: "kube-api-access-b5c95") pod "e4151c4f-b915-413a-8fb3-a1ae92cfa081" (UID: "e4151c4f-b915-413a-8fb3-a1ae92cfa081"). InnerVolumeSpecName "kube-api-access-b5c95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:41.139331 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.139312    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:41.139424 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.139333    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:41.139424 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.139347    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4151c4f-b915-413a-8fb3-a1ae92cfa081-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:41.139424 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.139355    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4151c4f-b915-413a-8fb3-a1ae92cfa081-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:41.139424 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.139364    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5c95\" (UniqueName: \"kubernetes.io/projected/e4151c4f-b915-413a-8fb3-a1ae92cfa081-kube-api-access-b5c95\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:41.199899 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.199860    2555 generic.go:356] "Generic (PLEG): container finished" podID="e4151c4f-b915-413a-8fb3-a1ae92cfa081" containerID="49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c" exitCode=0
Jul 28 20:16:41.200011 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.199923    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"
Jul 28 20:16:41.200011 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.199930    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q" event={"ID":"e4151c4f-b915-413a-8fb3-a1ae92cfa081","Type":"ContainerDied","Data":"49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c"}
Jul 28 20:16:41.200011 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.199964    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q" event={"ID":"e4151c4f-b915-413a-8fb3-a1ae92cfa081","Type":"ContainerDied","Data":"23dc67db78846ec12c65f12a6296abef42c716f4575196bbb19cd394f0b6ab2c"}
Jul 28 20:16:41.200011 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.199981    2555 scope.go:117] "RemoveContainer" containerID="49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c"
Jul 28 20:16:41.208785 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.208768    2555 scope.go:117] "RemoveContainer" containerID="49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c"
Jul 28 20:16:41.209053 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:16:41.209030    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c\": container with ID starting with 49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c not found: ID does not exist" containerID="49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c"
Jul 28 20:16:41.209140 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.209059    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c"} err="failed to get container status \"49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c\": rpc error: code = NotFound desc = could not find container \"49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c\": container with ID starting with 49de16aa0d45c6a396cd00f9c1e1b1234fdbd6dd7060568e0794612f78c9162c not found: ID does not exist"
Jul 28 20:16:41.223409 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.223390    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"]
Jul 28 20:16:41.227378 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.227354    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ab7491/test-model-ab7491-kserve-85d965cdcc-ljv9q"]
Jul 28 20:16:41.984672 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.984635    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee0e0f8-35ad-4a2d-897f-be057fb90c54" path="/var/lib/kubelet/pods/9ee0e0f8-35ad-4a2d-897f-be057fb90c54/volumes"
Jul 28 20:16:41.985069 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:41.985051    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4151c4f-b915-413a-8fb3-a1ae92cfa081" path="/var/lib/kubelet/pods/e4151c4f-b915-413a-8fb3-a1ae92cfa081/volumes"
Jul 28 20:16:42.877852 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:42.877800    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:42.977965 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:42.977939    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:43.160042 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:43.159972    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:43.379573 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:43.379547    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:45.503189 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:45.503098    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:16:54.353261 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.353220    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"]
Jul 28 20:16:54.353750 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.353513    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7" podUID="9cee0485-7fd1-4c4c-934a-15e55d3a3c92" containerName="maas-api" containerID="cri-o://fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639" gracePeriod=30
Jul 28 20:16:54.395588 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.395565    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:54.467608 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.467580    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"]
Jul 28 20:16:54.467861 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.467836    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97" podUID="7d4fa414-e18a-489c-bdd6-3b70d13b81c7" containerName="payload-processing" containerID="cri-o://59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a" gracePeriod=30
Jul 28 20:16:54.531713 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.531660    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:54.606107 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.606054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:16:54.636571 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.636279    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vfjw\" (UniqueName: \"kubernetes.io/projected/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-kube-api-access-2vfjw\") pod \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") "
Jul 28 20:16:54.636571 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.636350    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-openshift-service-ca\") pod \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") "
Jul 28 20:16:54.636571 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.636385    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls\") pod \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\" (UID: \"9cee0485-7fd1-4c4c-934a-15e55d3a3c92\") "
Jul 28 20:16:54.636571 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.636446    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:54.636998 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.636771    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9cee0485-7fd1-4c4c-934a-15e55d3a3c92" (UID: "9cee0485-7fd1-4c4c-934a-15e55d3a3c92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:16:54.638623 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.638594    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-kube-api-access-2vfjw" (OuterVolumeSpecName: "kube-api-access-2vfjw") pod "9cee0485-7fd1-4c4c-934a-15e55d3a3c92" (UID: "9cee0485-7fd1-4c4c-934a-15e55d3a3c92"). InnerVolumeSpecName "kube-api-access-2vfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:54.639030 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.639007    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9cee0485-7fd1-4c4c-934a-15e55d3a3c92" (UID: "9cee0485-7fd1-4c4c-934a-15e55d3a3c92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:16:54.712416 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.712393    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:16:54.733407 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.733382    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:54.737884 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.737868    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vfjw\" (UniqueName: \"kubernetes.io/projected/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-kube-api-access-2vfjw\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:54.737884 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.737884    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:54.738028 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.737909    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cee0485-7fd1-4c4c-934a-15e55d3a3c92-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:54.838667 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.838647    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd5gl\" (UniqueName: \"kubernetes.io/projected/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-kube-api-access-cd5gl\") pod \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\" (UID: \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\") "
Jul 28 20:16:54.838760 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.838679    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-plugins-config-volume\") pod \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\" (UID: \"7d4fa414-e18a-489c-bdd6-3b70d13b81c7\") "
Jul 28 20:16:54.839093 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.839075    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d4fa414-e18a-489c-bdd6-3b70d13b81c7" (UID: "7d4fa414-e18a-489c-bdd6-3b70d13b81c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:16:54.840583 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.840560    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-kube-api-access-cd5gl" (OuterVolumeSpecName: "kube-api-access-cd5gl") pod "7d4fa414-e18a-489c-bdd6-3b70d13b81c7" (UID: "7d4fa414-e18a-489c-bdd6-3b70d13b81c7"). InnerVolumeSpecName "kube-api-access-cd5gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:54.939205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.939152    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5gl\" (UniqueName: \"kubernetes.io/projected/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-kube-api-access-cd5gl\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:54.939205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:54.939173    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4fa414-e18a-489c-bdd6-3b70d13b81c7-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:16:55.176822 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.176795    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:16:55.265289 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.265219    2555 generic.go:356] "Generic (PLEG): container finished" podID="7d4fa414-e18a-489c-bdd6-3b70d13b81c7" containerID="59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a" exitCode=0
Jul 28 20:16:55.265289 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.265276    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"
Jul 28 20:16:55.265439 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.265315    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97" event={"ID":"7d4fa414-e18a-489c-bdd6-3b70d13b81c7","Type":"ContainerDied","Data":"59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a"}
Jul 28 20:16:55.265439 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.265375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97" event={"ID":"7d4fa414-e18a-489c-bdd6-3b70d13b81c7","Type":"ContainerDied","Data":"69e83d6404c888e5ec6b591b1e735f9d16255136341f6a8cc3e5792a9f52b1c4"}
Jul 28 20:16:55.265439 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.265400    2555 scope.go:117] "RemoveContainer" containerID="59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a"
Jul 28 20:16:55.266852 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.266829    2555 generic.go:356] "Generic (PLEG): container finished" podID="9cee0485-7fd1-4c4c-934a-15e55d3a3c92" containerID="fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639" exitCode=0
Jul 28 20:16:55.266958 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.266913    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7" event={"ID":"9cee0485-7fd1-4c4c-934a-15e55d3a3c92","Type":"ContainerDied","Data":"fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639"}
Jul 28 20:16:55.266958 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.266925    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"
Jul 28 20:16:55.266958 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.266950    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7" event={"ID":"9cee0485-7fd1-4c4c-934a-15e55d3a3c92","Type":"ContainerDied","Data":"e3c5381eda0c55e233926e9860a35c4d35f86fcef82dfc281ab48e09c5b4807c"}
Jul 28 20:16:55.274605 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.274577    2555 scope.go:117] "RemoveContainer" containerID="59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a"
Jul 28 20:16:55.274975 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:16:55.274946    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a\": container with ID starting with 59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a not found: ID does not exist" containerID="59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a"
Jul 28 20:16:55.275044 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.274988    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a"} err="failed to get container status \"59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a\": rpc error: code = NotFound desc = could not find container \"59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a\": container with ID starting with 59d87f3cfddb61e0f441f859555f2650c25595df5f04eecb25425169daa8040a not found: ID does not exist"
Jul 28 20:16:55.275044 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.275013    2555 scope.go:117] "RemoveContainer" containerID="fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639"
Jul 28 20:16:55.283392 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.283372    2555 scope.go:117] "RemoveContainer" containerID="fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639"
Jul 28 20:16:55.283720 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:16:55.283697    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639\": container with ID starting with fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639 not found: ID does not exist" containerID="fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639"
Jul 28 20:16:55.283823 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.283724    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639"} err="failed to get container status \"fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639\": rpc error: code = NotFound desc = could not find container \"fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639\": container with ID starting with fd9871030603a88a967a8374b66c9b25af85eabbd7f1c834bcff6aa8adc2a639 not found: ID does not exist"
Jul 28 20:16:55.303768 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.303744    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"]
Jul 28 20:16:55.306836 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.306811    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab7491-78657df6f8-2gjt7"]
Jul 28 20:16:55.332539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.332517    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"]
Jul 28 20:16:55.339830 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.339810    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ab7491-7dc88878d4-psb97"]
Jul 28 20:16:55.984391 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.984361    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d4fa414-e18a-489c-bdd6-3b70d13b81c7" path="/var/lib/kubelet/pods/7d4fa414-e18a-489c-bdd6-3b70d13b81c7/volumes"
Jul 28 20:16:55.984743 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:55.984719    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cee0485-7fd1-4c4c-934a-15e55d3a3c92" path="/var/lib/kubelet/pods/9cee0485-7fd1-4c4c-934a-15e55d3a3c92/volumes"
Jul 28 20:16:57.506285 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:16:57.506251    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:17:00.924663 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:00.924627    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:01.025951 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:01.025916    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:01.198568 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:01.198495    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:01.454534 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:01.454413    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:01.796071 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:01.796040    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:02.155657 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:02.155580    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:02.520671 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:02.520590    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:04.766125 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:04.766089    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:04.866561 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:04.866526    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:05.000103 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000071    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"]
Jul 28 20:17:05.000520 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000501    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e74d83-5953-4744-ba6f-92c2c7525bdf" containerName="maas-api"
Jul 28 20:17:05.000590 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000525    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e74d83-5953-4744-ba6f-92c2c7525bdf" containerName="maas-api"
Jul 28 20:17:05.000590 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000564    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cee0485-7fd1-4c4c-934a-15e55d3a3c92" containerName="maas-api"
Jul 28 20:17:05.000590 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000575    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cee0485-7fd1-4c4c-934a-15e55d3a3c92" containerName="maas-api"
Jul 28 20:17:05.000590 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000585    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee0e0f8-35ad-4a2d-897f-be057fb90c54" containerName="main"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000595    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee0e0f8-35ad-4a2d-897f-be057fb90c54" containerName="main"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000605    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4151c4f-b915-413a-8fb3-a1ae92cfa081" containerName="main"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000614    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4151c4f-b915-413a-8fb3-a1ae92cfa081" containerName="main"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000625    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d4fa414-e18a-489c-bdd6-3b70d13b81c7" containerName="payload-processing"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000634    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d4fa414-e18a-489c-bdd6-3b70d13b81c7" containerName="payload-processing"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000727    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee0e0f8-35ad-4a2d-897f-be057fb90c54" containerName="main"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000739    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d4fa414-e18a-489c-bdd6-3b70d13b81c7" containerName="payload-processing"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000746    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cee0485-7fd1-4c4c-934a-15e55d3a3c92" containerName="maas-api"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000755    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e74d83-5953-4744-ba6f-92c2c7525bdf" containerName="maas-api"
Jul 28 20:17:05.000783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.000764    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4151c4f-b915-413a-8fb3-a1ae92cfa081" containerName="main"
Jul 28 20:17:05.003565 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.003546    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.006636 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.006608    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xsbnd\""
Jul 28 20:17:05.006780 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.006636    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:17:05.006780 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.006635    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a5ccdf-serving-cert\""
Jul 28 20:17:05.007576 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.007558    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:17:05.014639 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.014614    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"]
Jul 28 20:17:05.038999 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.038944    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:05.107277 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.107250    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnhs\" (UniqueName: \"kubernetes.io/projected/1886539d-4bde-4e15-b57a-f95103905404-kube-api-access-4hnhs\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.107386 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.107286    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1886539d-4bde-4e15-b57a-f95103905404-maas-api-tls\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.107386 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.107318    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1886539d-4bde-4e15-b57a-f95103905404-openshift-service-ca\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.208299 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.208272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1886539d-4bde-4e15-b57a-f95103905404-maas-api-tls\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.208395 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.208312    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1886539d-4bde-4e15-b57a-f95103905404-openshift-service-ca\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.208395 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.208370    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnhs\" (UniqueName: \"kubernetes.io/projected/1886539d-4bde-4e15-b57a-f95103905404-kube-api-access-4hnhs\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.208998 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.208978    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1886539d-4bde-4e15-b57a-f95103905404-openshift-service-ca\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.210619 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.210594    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1886539d-4bde-4e15-b57a-f95103905404-maas-api-tls\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.222484 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.222463    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnhs\" (UniqueName: \"kubernetes.io/projected/1886539d-4bde-4e15-b57a-f95103905404-kube-api-access-4hnhs\") pod \"maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.232055 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.232035    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:05.315083 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.315057    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:05.465539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.465514    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"]
Jul 28 20:17:05.468762 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:17:05.468734    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1886539d_4bde_4e15_b57a_f95103905404.slice/crio-5bc017fc3473a5c85b030a5a8ed754f68936c656cf13cc8dac497ca72e8cdc0a WatchSource:0}: Error finding container 5bc017fc3473a5c85b030a5a8ed754f68936c656cf13cc8dac497ca72e8cdc0a: Status 404 returned error can't find the container with id 5bc017fc3473a5c85b030a5a8ed754f68936c656cf13cc8dac497ca72e8cdc0a
Jul 28 20:17:05.497603 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.497579    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:05.597860 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.597800    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:05.658577 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.658546    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"]
Jul 28 20:17:05.662704 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.662685    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:05.668713 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.668694    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a5ccdf\""
Jul 28 20:17:05.669282 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.669094    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a5ccdf-dockercfg-s5jjf\""
Jul 28 20:17:05.684470 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.684451    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"]
Jul 28 20:17:05.713219 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.713191    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl\" (UID: \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:05.713303 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.713245    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxp66\" (UniqueName: \"kubernetes.io/projected/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-kube-api-access-hxp66\") pod \"payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl\" (UID: \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:05.814304 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.814277    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl\" (UID: \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:05.814597 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.814342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxp66\" (UniqueName: \"kubernetes.io/projected/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-kube-api-access-hxp66\") pod \"payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl\" (UID: \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:05.815039 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.815023    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl\" (UID: \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:05.832225 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.832200    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxp66\" (UniqueName: \"kubernetes.io/projected/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-kube-api-access-hxp66\") pod \"payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl\" (UID: \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:05.972714 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:05.972687    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:06.098602 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.098575    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"]
Jul 28 20:17:06.100942 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:17:06.100916    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d1c4a2_e14e_47e6_8d22_8626fa4074c5.slice/crio-18954033a8be53fdf9399b605e080de609179d531e9b6659d8f964ec81eefb17 WatchSource:0}: Error finding container 18954033a8be53fdf9399b605e080de609179d531e9b6659d8f964ec81eefb17: Status 404 returned error can't find the container with id 18954033a8be53fdf9399b605e080de609179d531e9b6659d8f964ec81eefb17
Jul 28 20:17:06.316077 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.316044    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl" event={"ID":"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5","Type":"ContainerStarted","Data":"f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59"}
Jul 28 20:17:06.316214 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.316103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl" event={"ID":"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5","Type":"ContainerStarted","Data":"18954033a8be53fdf9399b605e080de609179d531e9b6659d8f964ec81eefb17"}
Jul 28 20:17:06.316214 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.316131    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:06.317481 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.317456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c" event={"ID":"1886539d-4bde-4e15-b57a-f95103905404","Type":"ContainerStarted","Data":"209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546"}
Jul 28 20:17:06.317584 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.317498    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c" event={"ID":"1886539d-4bde-4e15-b57a-f95103905404","Type":"ContainerStarted","Data":"5bc017fc3473a5c85b030a5a8ed754f68936c656cf13cc8dac497ca72e8cdc0a"}
Jul 28 20:17:06.317632 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.317607    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:06.333717 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.333677    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl" podStartSLOduration=1.333662038 podStartE2EDuration="1.333662038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:06.332183662 +0000 UTC m=+2444.792942263" watchObservedRunningTime="2026-07-28 20:17:06.333662038 +0000 UTC m=+2444.794420634"
Jul 28 20:17:06.349032 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:06.348995    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c" podStartSLOduration=1.9224919900000002 podStartE2EDuration="2.348981251s" totalImagesPullingTime="426.489261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:06.347598557 +0000 UTC m=+2444.808357156" watchObservedRunningTime="2026-07-28 20:17:06.348981251 +0000 UTC m=+2444.809739850"
Jul 28 20:17:11.097852 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:11.097804    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:17:12.329071 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:12.329038    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:17:17.325601 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:17.325562    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:17:29.671862 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:29.671822    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:30.145226 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:30.145187    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:30.574096 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:30.574058    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:30.666261 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:30.666235    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:33.220735 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.220658    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:33.307162 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.307128    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:33.308028 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.308007    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"]
Jul 28 20:17:33.311016 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.311000    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.313659 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.313631    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bf09e7-serving-cert\""
Jul 28 20:17:33.320679 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.320660    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"]
Jul 28 20:17:33.414059 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.414033    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:33.419237 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.419208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsf8\" (UniqueName: \"kubernetes.io/projected/9a02e136-6e10-4994-9d4d-81b2f51f1723-kube-api-access-9bsf8\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.419347 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.419255    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a02e136-6e10-4994-9d4d-81b2f51f1723-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.419347 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.419282    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a02e136-6e10-4994-9d4d-81b2f51f1723-maas-api-tls\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.516327 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.516256    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:33.520304 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.520282    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsf8\" (UniqueName: \"kubernetes.io/projected/9a02e136-6e10-4994-9d4d-81b2f51f1723-kube-api-access-9bsf8\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.520409 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.520321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a02e136-6e10-4994-9d4d-81b2f51f1723-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.520409 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.520370    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a02e136-6e10-4994-9d4d-81b2f51f1723-maas-api-tls\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.521005 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.520987    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a02e136-6e10-4994-9d4d-81b2f51f1723-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.522637 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.522606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a02e136-6e10-4994-9d4d-81b2f51f1723-maas-api-tls\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.533865 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.533841    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsf8\" (UniqueName: \"kubernetes.io/projected/9a02e136-6e10-4994-9d4d-81b2f51f1723-kube-api-access-9bsf8\") pod \"maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.620871 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.620849    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:33.744653 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.744609    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"]
Jul 28 20:17:33.746775 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:17:33.746751    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a02e136_6e10_4994_9d4d_81b2f51f1723.slice/crio-80fd4a3c5ecbb19e8a54aa950e47f656444df8e66fc96c64670dbc6bb6ab60dc WatchSource:0}: Error finding container 80fd4a3c5ecbb19e8a54aa950e47f656444df8e66fc96c64670dbc6bb6ab60dc: Status 404 returned error can't find the container with id 80fd4a3c5ecbb19e8a54aa950e47f656444df8e66fc96c64670dbc6bb6ab60dc
Jul 28 20:17:33.898450 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.898422    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:33.997566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:33.997541    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:17:34.003273 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.003251    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"]
Jul 28 20:17:34.007670 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.007655    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:34.010301 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.010279    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bf09e7\""
Jul 28 20:17:34.016973 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.016952    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"]
Jul 28 20:17:34.125144 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.125115    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w\" (UID: \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:34.125237 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.125143    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfpm\" (UniqueName: \"kubernetes.io/projected/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-kube-api-access-pjfpm\") pod \"payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w\" (UID: \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:34.225493 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.225440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w\" (UID: \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:34.225493 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.225470    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfpm\" (UniqueName: \"kubernetes.io/projected/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-kube-api-access-pjfpm\") pod \"payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w\" (UID: \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:34.226108 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.226090    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w\" (UID: \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:34.233094 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.233076    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfpm\" (UniqueName: \"kubernetes.io/projected/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-kube-api-access-pjfpm\") pod \"payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w\" (UID: \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:34.318337 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.318306    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:34.446654 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.446625    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh" event={"ID":"9a02e136-6e10-4994-9d4d-81b2f51f1723","Type":"ContainerStarted","Data":"80fd4a3c5ecbb19e8a54aa950e47f656444df8e66fc96c64670dbc6bb6ab60dc"}
Jul 28 20:17:34.451345 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:34.451324    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"]
Jul 28 20:17:34.459960 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:17:34.459936    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc4fc7f_56fd_45fb_827e_9aa2b6651d1e.slice/crio-6a59b75a60b1bf8839243e3ba11f1c31973e1d00ed1d2d86897ce08f671a8e3e WatchSource:0}: Error finding container 6a59b75a60b1bf8839243e3ba11f1c31973e1d00ed1d2d86897ce08f671a8e3e: Status 404 returned error can't find the container with id 6a59b75a60b1bf8839243e3ba11f1c31973e1d00ed1d2d86897ce08f671a8e3e
Jul 28 20:17:35.452559 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:35.452524    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w" event={"ID":"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e","Type":"ContainerStarted","Data":"7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c"}
Jul 28 20:17:35.453006 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:35.452577    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w" event={"ID":"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e","Type":"ContainerStarted","Data":"6a59b75a60b1bf8839243e3ba11f1c31973e1d00ed1d2d86897ce08f671a8e3e"}
Jul 28 20:17:35.453006 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:35.452604    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:35.454058 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:35.454034    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh" event={"ID":"9a02e136-6e10-4994-9d4d-81b2f51f1723","Type":"ContainerStarted","Data":"59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a"}
Jul 28 20:17:35.454174 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:35.454085    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:35.471407 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:35.471367    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w" podStartSLOduration=2.4713525880000002 podStartE2EDuration="2.471352588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:35.469942681 +0000 UTC m=+2473.930701280" watchObservedRunningTime="2026-07-28 20:17:35.471352588 +0000 UTC m=+2473.932111213"
Jul 28 20:17:35.487294 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:35.487252    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh" podStartSLOduration=1.827329005 podStartE2EDuration="2.487239125s" totalImagesPullingTime="659.91012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:35.485990875 +0000 UTC m=+2473.946749475" watchObservedRunningTime="2026-07-28 20:17:35.487239125 +0000 UTC m=+2473.947997723"
Jul 28 20:17:35.999378 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:35.999351    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:17:41.462907 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:41.462863    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:17:46.460206 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:46.460171    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:17:58.261542 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:17:58.261500    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:03.595520 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:03.595481    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:13.755772 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:13.755735    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:16.630951 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.630915    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"]
Jul 28 20:18:16.631468 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.631195    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c" podUID="1886539d-4bde-4e15-b57a-f95103905404" containerName="maas-api" containerID="cri-o://209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546" gracePeriod=30
Jul 28 20:18:16.666463 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.666441    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:16.700839 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.700815    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"]
Jul 28 20:18:16.701020 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.700998    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl" podUID="d1d1c4a2-e14e-47e6-8d22-8626fa4074c5" containerName="payload-pre-processing" containerID="cri-o://f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59" gracePeriod=30
Jul 28 20:18:16.769472 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.769437    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:16.854963 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.854915    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:16.917473 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.917438    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:16.949122 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.949099    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:18:16.969554 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:16.969221    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:18:17.034778 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.034750    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1886539d-4bde-4e15-b57a-f95103905404-openshift-service-ca\") pod \"1886539d-4bde-4e15-b57a-f95103905404\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") "
Jul 28 20:18:17.034928 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.034781    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hnhs\" (UniqueName: \"kubernetes.io/projected/1886539d-4bde-4e15-b57a-f95103905404-kube-api-access-4hnhs\") pod \"1886539d-4bde-4e15-b57a-f95103905404\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") "
Jul 28 20:18:17.034928 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.034809    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxp66\" (UniqueName: \"kubernetes.io/projected/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-kube-api-access-hxp66\") pod \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\" (UID: \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\") "
Jul 28 20:18:17.034928 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.034842    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-plugins-config-volume\") pod \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\" (UID: \"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5\") "
Jul 28 20:18:17.034928 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.034873    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1886539d-4bde-4e15-b57a-f95103905404-maas-api-tls\") pod \"1886539d-4bde-4e15-b57a-f95103905404\" (UID: \"1886539d-4bde-4e15-b57a-f95103905404\") "
Jul 28 20:18:17.035198 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.035170    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1886539d-4bde-4e15-b57a-f95103905404-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1886539d-4bde-4e15-b57a-f95103905404" (UID: "1886539d-4bde-4e15-b57a-f95103905404"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:18:17.035269 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.035212    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1d1c4a2-e14e-47e6-8d22-8626fa4074c5" (UID: "d1d1c4a2-e14e-47e6-8d22-8626fa4074c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:18:17.036875 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.036853    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1886539d-4bde-4e15-b57a-f95103905404-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1886539d-4bde-4e15-b57a-f95103905404" (UID: "1886539d-4bde-4e15-b57a-f95103905404"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:18:17.036967 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.036870    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-kube-api-access-hxp66" (OuterVolumeSpecName: "kube-api-access-hxp66") pod "d1d1c4a2-e14e-47e6-8d22-8626fa4074c5" (UID: "d1d1c4a2-e14e-47e6-8d22-8626fa4074c5"). InnerVolumeSpecName "kube-api-access-hxp66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:17.036967 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.036926    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1886539d-4bde-4e15-b57a-f95103905404-kube-api-access-4hnhs" (OuterVolumeSpecName: "kube-api-access-4hnhs") pod "1886539d-4bde-4e15-b57a-f95103905404" (UID: "1886539d-4bde-4e15-b57a-f95103905404"). InnerVolumeSpecName "kube-api-access-4hnhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:17.136325 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.136292    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1886539d-4bde-4e15-b57a-f95103905404-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:17.136325 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.136323    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hnhs\" (UniqueName: \"kubernetes.io/projected/1886539d-4bde-4e15-b57a-f95103905404-kube-api-access-4hnhs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:17.136325 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.136333    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxp66\" (UniqueName: \"kubernetes.io/projected/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-kube-api-access-hxp66\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:17.136483 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.136342    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:17.136483 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.136351    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1886539d-4bde-4e15-b57a-f95103905404-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:17.225607 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.225534    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:17.635038 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.635004    2555 generic.go:356] "Generic (PLEG): container finished" podID="1886539d-4bde-4e15-b57a-f95103905404" containerID="209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546" exitCode=0
Jul 28 20:18:17.635474 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.635086    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"
Jul 28 20:18:17.635474 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.635097    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c" event={"ID":"1886539d-4bde-4e15-b57a-f95103905404","Type":"ContainerDied","Data":"209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546"}
Jul 28 20:18:17.635474 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.635163    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c" event={"ID":"1886539d-4bde-4e15-b57a-f95103905404","Type":"ContainerDied","Data":"5bc017fc3473a5c85b030a5a8ed754f68936c656cf13cc8dac497ca72e8cdc0a"}
Jul 28 20:18:17.635474 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.635193    2555 scope.go:117] "RemoveContainer" containerID="209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546"
Jul 28 20:18:17.636717 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.636640    2555 generic.go:356] "Generic (PLEG): container finished" podID="d1d1c4a2-e14e-47e6-8d22-8626fa4074c5" containerID="f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59" exitCode=0
Jul 28 20:18:17.636717 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.636699    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl" event={"ID":"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5","Type":"ContainerDied","Data":"f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59"}
Jul 28 20:18:17.636943 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.636742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl" event={"ID":"d1d1c4a2-e14e-47e6-8d22-8626fa4074c5","Type":"ContainerDied","Data":"18954033a8be53fdf9399b605e080de609179d531e9b6659d8f964ec81eefb17"}
Jul 28 20:18:17.636943 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.636747    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"
Jul 28 20:18:17.652174 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.652157    2555 scope.go:117] "RemoveContainer" containerID="209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546"
Jul 28 20:18:17.652470 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:18:17.652449    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546\": container with ID starting with 209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546 not found: ID does not exist" containerID="209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546"
Jul 28 20:18:17.652540 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.652474    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546"} err="failed to get container status \"209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546\": rpc error: code = NotFound desc = could not find container \"209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546\": container with ID starting with 209e6bd8a6ac9c1dff6b172c28fd74b02aea02bb68318d1ab2c5fc0f0e82a546 not found: ID does not exist"
Jul 28 20:18:17.652540 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.652491    2555 scope.go:117] "RemoveContainer" containerID="f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59"
Jul 28 20:18:17.660189 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.660170    2555 scope.go:117] "RemoveContainer" containerID="f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59"
Jul 28 20:18:17.660456 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:18:17.660435    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59\": container with ID starting with f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59 not found: ID does not exist" containerID="f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59"
Jul 28 20:18:17.660514 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.660463    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59"} err="failed to get container status \"f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59\": rpc error: code = NotFound desc = could not find container \"f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59\": container with ID starting with f6f8addff543bea93f6b742fba8811439e8972abf8c9532a49df013d184b8a59 not found: ID does not exist"
Jul 28 20:18:17.666594 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.666569    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"]
Jul 28 20:18:17.670212 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.670189    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a5ccdf-5d59ff8b8d-mc25c"]
Jul 28 20:18:17.684767 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.684743    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"]
Jul 28 20:18:17.689158 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.689139    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a5ccdf-756f995668-m7bfl"]
Jul 28 20:18:17.985769 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.985688    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1886539d-4bde-4e15-b57a-f95103905404" path="/var/lib/kubelet/pods/1886539d-4bde-4e15-b57a-f95103905404/volumes"
Jul 28 20:18:17.986172 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:17.986148    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d1c4a2-e14e-47e6-8d22-8626fa4074c5" path="/var/lib/kubelet/pods/d1d1c4a2-e14e-47e6-8d22-8626fa4074c5/volumes"
Jul 28 20:18:22.405539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:22.405503    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:18:23.059485 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:23.059454    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:23.161217 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:23.161189    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:23.350372 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:23.350296    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:23.575317 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:23.575288    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:39.077270 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.077233    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"]
Jul 28 20:18:39.077838 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.077512    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh" podUID="9a02e136-6e10-4994-9d4d-81b2f51f1723" containerName="maas-api" containerID="cri-o://59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a" gracePeriod=30
Jul 28 20:18:39.106207 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.106173    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:39.139556 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.139405    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"]
Jul 28 20:18:39.139701 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.139648    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w" podUID="9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e" containerName="payload-pre-processing" containerID="cri-o://7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c" gracePeriod=30
Jul 28 20:18:39.206689 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.206646    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:39.274432 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.274404    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:39.329658 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.329573    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:39.373691 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.373669    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:18:39.404239 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.404217    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:18:39.496101 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.496067    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a02e136-6e10-4994-9d4d-81b2f51f1723-maas-api-tls\") pod \"9a02e136-6e10-4994-9d4d-81b2f51f1723\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") "
Jul 28 20:18:39.496225 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.496135    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bsf8\" (UniqueName: \"kubernetes.io/projected/9a02e136-6e10-4994-9d4d-81b2f51f1723-kube-api-access-9bsf8\") pod \"9a02e136-6e10-4994-9d4d-81b2f51f1723\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") "
Jul 28 20:18:39.496225 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.496181    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a02e136-6e10-4994-9d4d-81b2f51f1723-openshift-service-ca\") pod \"9a02e136-6e10-4994-9d4d-81b2f51f1723\" (UID: \"9a02e136-6e10-4994-9d4d-81b2f51f1723\") "
Jul 28 20:18:39.496225 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.496215    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjfpm\" (UniqueName: \"kubernetes.io/projected/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-kube-api-access-pjfpm\") pod \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\" (UID: \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\") "
Jul 28 20:18:39.496364 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.496239    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-plugins-config-volume\") pod \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\" (UID: \"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e\") "
Jul 28 20:18:39.496616 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.496584    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a02e136-6e10-4994-9d4d-81b2f51f1723-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a02e136-6e10-4994-9d4d-81b2f51f1723" (UID: "9a02e136-6e10-4994-9d4d-81b2f51f1723"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:18:39.496735 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.496644    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e" (UID: "9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:18:39.498216 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.498184    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-kube-api-access-pjfpm" (OuterVolumeSpecName: "kube-api-access-pjfpm") pod "9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e" (UID: "9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e"). InnerVolumeSpecName "kube-api-access-pjfpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:39.498216 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.498209    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a02e136-6e10-4994-9d4d-81b2f51f1723-kube-api-access-9bsf8" (OuterVolumeSpecName: "kube-api-access-9bsf8") pod "9a02e136-6e10-4994-9d4d-81b2f51f1723" (UID: "9a02e136-6e10-4994-9d4d-81b2f51f1723"). InnerVolumeSpecName "kube-api-access-9bsf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:39.498344 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.498220    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a02e136-6e10-4994-9d4d-81b2f51f1723-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a02e136-6e10-4994-9d4d-81b2f51f1723" (UID: "9a02e136-6e10-4994-9d4d-81b2f51f1723"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:18:39.597365 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.597301    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjfpm\" (UniqueName: \"kubernetes.io/projected/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-kube-api-access-pjfpm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:39.597365 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.597332    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:39.597365 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.597343    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a02e136-6e10-4994-9d4d-81b2f51f1723-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:39.597365 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.597352    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bsf8\" (UniqueName: \"kubernetes.io/projected/9a02e136-6e10-4994-9d4d-81b2f51f1723-kube-api-access-9bsf8\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:39.597365 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.597360    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a02e136-6e10-4994-9d4d-81b2f51f1723-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:18:39.723655 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.723622    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:39.738479 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.738451    2555 generic.go:356] "Generic (PLEG): container finished" podID="9a02e136-6e10-4994-9d4d-81b2f51f1723" containerID="59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a" exitCode=0
Jul 28 20:18:39.738573 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.738500    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"
Jul 28 20:18:39.738573 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.738541    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh" event={"ID":"9a02e136-6e10-4994-9d4d-81b2f51f1723","Type":"ContainerDied","Data":"59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a"}
Jul 28 20:18:39.738689 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.738591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh" event={"ID":"9a02e136-6e10-4994-9d4d-81b2f51f1723","Type":"ContainerDied","Data":"80fd4a3c5ecbb19e8a54aa950e47f656444df8e66fc96c64670dbc6bb6ab60dc"}
Jul 28 20:18:39.738689 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.738621    2555 scope.go:117] "RemoveContainer" containerID="59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a"
Jul 28 20:18:39.740026 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.740005    2555 generic.go:356] "Generic (PLEG): container finished" podID="9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e" containerID="7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c" exitCode=0
Jul 28 20:18:39.740108 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.740088    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"
Jul 28 20:18:39.740194 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.740088    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w" event={"ID":"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e","Type":"ContainerDied","Data":"7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c"}
Jul 28 20:18:39.740268 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.740232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w" event={"ID":"9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e","Type":"ContainerDied","Data":"6a59b75a60b1bf8839243e3ba11f1c31973e1d00ed1d2d86897ce08f671a8e3e"}
Jul 28 20:18:39.748390 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.748350    2555 scope.go:117] "RemoveContainer" containerID="59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a"
Jul 28 20:18:39.748667 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:18:39.748643    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a\": container with ID starting with 59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a not found: ID does not exist" containerID="59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a"
Jul 28 20:18:39.748746 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.748677    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a"} err="failed to get container status \"59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a\": rpc error: code = NotFound desc = could not find container \"59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a\": container with ID starting with 59390a09fd1f1f403e35764ddf63a1ab173a32da03f9a93d823b55615f70ef0a not found: ID does not exist"
Jul 28 20:18:39.748746 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.748698    2555 scope.go:117] "RemoveContainer" containerID="7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c"
Jul 28 20:18:39.756358 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.756342    2555 scope.go:117] "RemoveContainer" containerID="7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c"
Jul 28 20:18:39.756612 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:18:39.756594    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c\": container with ID starting with 7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c not found: ID does not exist" containerID="7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c"
Jul 28 20:18:39.756664 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.756620    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c"} err="failed to get container status \"7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c\": rpc error: code = NotFound desc = could not find container \"7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c\": container with ID starting with 7d95f5e89f32469031f6a9efd6c20e436aa73159135c9ca1eac492964126e13c not found: ID does not exist"
Jul 28 20:18:39.760967 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.760946    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"]
Jul 28 20:18:39.765128 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.765108    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf09e7-54fb5f5458-d24fh"]
Jul 28 20:18:39.776596 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.776575    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"]
Jul 28 20:18:39.781007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.780984    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-bf09e7-5f487f7cc6-nhb4w"]
Jul 28 20:18:39.983803 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.983750    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a02e136-6e10-4994-9d4d-81b2f51f1723" path="/var/lib/kubelet/pods/9a02e136-6e10-4994-9d4d-81b2f51f1723/volumes"
Jul 28 20:18:39.984145 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:39.984127    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e" path="/var/lib/kubelet/pods/9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e/volumes"
Jul 28 20:18:43.999993 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:43.999954    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:18:45.627320 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:45.627283    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:45.695462 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:45.695432    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:45.893187 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:45.893116    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:46.104607 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:46.104576    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:46.448311 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:46.448277    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:46.870648 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:46.870617    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:47.451824 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:47.451795    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:50.246512 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.246485    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:50.344944 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.344912    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:50.496628 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496546    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"]
Jul 28 20:18:50.496853 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496838    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d1c4a2-e14e-47e6-8d22-8626fa4074c5" containerName="payload-pre-processing"
Jul 28 20:18:50.496973 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496854    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d1c4a2-e14e-47e6-8d22-8626fa4074c5" containerName="payload-pre-processing"
Jul 28 20:18:50.496973 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496864    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1886539d-4bde-4e15-b57a-f95103905404" containerName="maas-api"
Jul 28 20:18:50.496973 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496870    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1886539d-4bde-4e15-b57a-f95103905404" containerName="maas-api"
Jul 28 20:18:50.496973 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496877    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e" containerName="payload-pre-processing"
Jul 28 20:18:50.496973 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496883    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e" containerName="payload-pre-processing"
Jul 28 20:18:50.496973 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496923    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a02e136-6e10-4994-9d4d-81b2f51f1723" containerName="maas-api"
Jul 28 20:18:50.496973 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496929    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a02e136-6e10-4994-9d4d-81b2f51f1723" containerName="maas-api"
Jul 28 20:18:50.497282 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496979    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d1c4a2-e14e-47e6-8d22-8626fa4074c5" containerName="payload-pre-processing"
Jul 28 20:18:50.497282 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496987    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a02e136-6e10-4994-9d4d-81b2f51f1723" containerName="maas-api"
Jul 28 20:18:50.497282 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.496995    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1886539d-4bde-4e15-b57a-f95103905404" containerName="maas-api"
Jul 28 20:18:50.497282 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.497000    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc4fc7f-56fd-45fb-827e-9aa2b6651d1e" containerName="payload-pre-processing"
Jul 28 20:18:50.501369 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.501354    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.504203 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.504183    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:18:50.504494 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.504479    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a1e1ef-serving-cert\""
Jul 28 20:18:50.504592 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.504531    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xsbnd\""
Jul 28 20:18:50.505296 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.505281    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:18:50.519204 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.519184    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"]
Jul 28 20:18:50.533116 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.533091    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:50.578708 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.578681    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.578817 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.578750    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.578817 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.578787    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgk85\" (UniqueName: \"kubernetes.io/projected/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-kube-api-access-wgk85\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.679944 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.679917    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.680035 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.679953    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgk85\" (UniqueName: \"kubernetes.io/projected/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-kube-api-access-wgk85\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.680035 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.680000    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.680145 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:18:50.680096    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-serving-cert: secret "maas-api-e2e-shared-a-a1e1ef-serving-cert" not found
Jul 28 20:18:50.680199 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:18:50.680157    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls podName:d2bc1188-8ae2-423b-8c88-fb7fc1759eb9 nodeName:}" failed. No retries permitted until 2026-07-28 20:18:51.18014078 +0000 UTC m=+2549.640899357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls") pod "maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk" (UID: "d2bc1188-8ae2-423b-8c88-fb7fc1759eb9") : secret "maas-api-e2e-shared-a-a1e1ef-serving-cert" not found
Jul 28 20:18:50.680519 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.680501    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.690923 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.690899    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgk85\" (UniqueName: \"kubernetes.io/projected/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-kube-api-access-wgk85\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:50.968324 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:50.968297    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:51.067680 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.067648    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:51.155242 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.155203    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:18:51.162591 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.162568    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"]
Jul 28 20:18:51.165881 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.165866    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.168360 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.168338    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-a1e1ef\""
Jul 28 20:18:51.168463 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.168395    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-a1e1ef-dockercfg-l7plf\""
Jul 28 20:18:51.175196 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.175176    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"]
Jul 28 20:18:51.184819 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.184797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:51.187114 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.187090    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls\") pod \"maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:51.285679 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.285614    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4fccce-07eb-4608-9704-bc4fd3071325-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv\" (UID: \"9b4fccce-07eb-4608-9704-bc4fd3071325\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.285679 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.285646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n2gt\" (UniqueName: \"kubernetes.io/projected/9b4fccce-07eb-4608-9704-bc4fd3071325-kube-api-access-8n2gt\") pod \"payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv\" (UID: \"9b4fccce-07eb-4608-9704-bc4fd3071325\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.386355 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.386327    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4fccce-07eb-4608-9704-bc4fd3071325-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv\" (UID: \"9b4fccce-07eb-4608-9704-bc4fd3071325\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.386459 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.386359    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n2gt\" (UniqueName: \"kubernetes.io/projected/9b4fccce-07eb-4608-9704-bc4fd3071325-kube-api-access-8n2gt\") pod \"payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv\" (UID: \"9b4fccce-07eb-4608-9704-bc4fd3071325\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.386985 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.386969    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4fccce-07eb-4608-9704-bc4fd3071325-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv\" (UID: \"9b4fccce-07eb-4608-9704-bc4fd3071325\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.395489 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.395470    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n2gt\" (UniqueName: \"kubernetes.io/projected/9b4fccce-07eb-4608-9704-bc4fd3071325-kube-api-access-8n2gt\") pod \"payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv\" (UID: \"9b4fccce-07eb-4608-9704-bc4fd3071325\") " pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.411320 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.411299    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:51.476781 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.476730    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.562361 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.557919    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"]
Jul 28 20:18:51.634176 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.634154    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"]
Jul 28 20:18:51.636544 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:18:51.636514    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4fccce_07eb_4608_9704_bc4fd3071325.slice/crio-13951e0fd8eefd229091105634c49f61d47a7c43a52ae367433e0eaf5dbe1d61 WatchSource:0}: Error finding container 13951e0fd8eefd229091105634c49f61d47a7c43a52ae367433e0eaf5dbe1d61: Status 404 returned error can't find the container with id 13951e0fd8eefd229091105634c49f61d47a7c43a52ae367433e0eaf5dbe1d61
Jul 28 20:18:51.797372 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.797340    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv" event={"ID":"9b4fccce-07eb-4608-9704-bc4fd3071325","Type":"ContainerStarted","Data":"33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96"}
Jul 28 20:18:51.797511 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.797387    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv" event={"ID":"9b4fccce-07eb-4608-9704-bc4fd3071325","Type":"ContainerStarted","Data":"13951e0fd8eefd229091105634c49f61d47a7c43a52ae367433e0eaf5dbe1d61"}
Jul 28 20:18:51.797511 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.797450    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:18:51.798560 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.798536    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk" event={"ID":"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9","Type":"ContainerStarted","Data":"cb32c9e2ff59de172ccf174e08e8666de9938bf3a4213da5c827cdb16719aafa"}
Jul 28 20:18:51.815187 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:51.815114    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv" podStartSLOduration=0.815100721 podStartE2EDuration="815.100721ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:18:51.814405652 +0000 UTC m=+2550.275164252" watchObservedRunningTime="2026-07-28 20:18:51.815100721 +0000 UTC m=+2550.275859338"
Jul 28 20:18:52.804200 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:52.804164    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk" event={"ID":"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9","Type":"ContainerStarted","Data":"fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23"}
Jul 28 20:18:52.804597 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:52.804229    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:18:52.822761 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:52.822716    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk" podStartSLOduration=2.396101076 podStartE2EDuration="2.82270064s" totalImagesPullingTime="426.599564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:18:52.821155291 +0000 UTC m=+2551.281913890" watchObservedRunningTime="2026-07-28 20:18:52.82270064 +0000 UTC m=+2551.283459239"
Jul 28 20:18:54.995297 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:54.995255    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:18:58.814933 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:18:58.814901    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:19:02.806216 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:02.806186    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:19:15.269019 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:15.268983    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:15.612831 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:15.612800    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:15.938882 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:15.938802    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:16.260769 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:16.260701    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:18.962191 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:18.962156    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:19.055795 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.055745    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:19.056469 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.056447    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"]
Jul 28 20:19:19.059787 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.059768    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.062654 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.062635    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-47f87c-serving-cert\""
Jul 28 20:19:19.073808 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.073785    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"]
Jul 28 20:19:19.132485 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.132468    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:19.179103 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.179078    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.179190 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.179110    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-openshift-service-ca\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.179190 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.179135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsn7b\" (UniqueName: \"kubernetes.io/projected/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-kube-api-access-tsn7b\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.279969 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.279910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.279969 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.279943    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-openshift-service-ca\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.279969 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.279968    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsn7b\" (UniqueName: \"kubernetes.io/projected/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-kube-api-access-tsn7b\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.280206 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:19:19.280067    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-serving-cert: secret "maas-api-e2e-shared-b-47f87c-serving-cert" not found
Jul 28 20:19:19.280206 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:19:19.280134    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls podName:8eafb2d8-b01e-4a4b-b407-a6659afe8f31 nodeName:}" failed. No retries permitted until 2026-07-28 20:19:19.780113303 +0000 UTC m=+2578.240871880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls") pod "maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh" (UID: "8eafb2d8-b01e-4a4b-b407-a6659afe8f31") : secret "maas-api-e2e-shared-b-47f87c-serving-cert" not found
Jul 28 20:19:19.280577 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.280560    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-openshift-service-ca\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.291872 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.291843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsn7b\" (UniqueName: \"kubernetes.io/projected/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-kube-api-access-tsn7b\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.692468 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.692429    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:19.784166 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.784137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.786602 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.786573    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls\") pod \"maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:19.843129 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.843105    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:19.943833 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.943769    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:19.971217 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:19.971196    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:20.105248 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:20.105226    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"]
Jul 28 20:19:20.107881 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:19:20.107848    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eafb2d8_b01e_4a4b_b407_a6659afe8f31.slice/crio-4b8e7c4583b6be8958683edbbb2a5e7cde1caa2ae86dff7d266e2d96c0b15dab WatchSource:0}: Error finding container 4b8e7c4583b6be8958683edbbb2a5e7cde1caa2ae86dff7d266e2d96c0b15dab: Status 404 returned error can't find the container with id 4b8e7c4583b6be8958683edbbb2a5e7cde1caa2ae86dff7d266e2d96c0b15dab
Jul 28 20:19:20.109290 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:20.109271    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:19:20.930703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:20.930659    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh" event={"ID":"8eafb2d8-b01e-4a4b-b407-a6659afe8f31","Type":"ContainerStarted","Data":"b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2"}
Jul 28 20:19:20.930877 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:20.930724    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh" event={"ID":"8eafb2d8-b01e-4a4b-b407-a6659afe8f31","Type":"ContainerStarted","Data":"4b8e7c4583b6be8958683edbbb2a5e7cde1caa2ae86dff7d266e2d96c0b15dab"}
Jul 28 20:19:20.930877 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:20.930753    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:20.953215 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:20.953171    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh" podStartSLOduration=1.594834418 podStartE2EDuration="1.953155561s" totalImagesPullingTime="358.321143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:20.951650536 +0000 UTC m=+2579.412409135" watchObservedRunningTime="2026-07-28 20:19:20.953155561 +0000 UTC m=+2579.413914160"
Jul 28 20:19:23.679617 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:23.679584    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:26.946623 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:26.946590    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:19:27.013643 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:27.013615    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:19:30.607737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:30.607693    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:43.942802 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:43.942706    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:53.498608 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.498570    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"]
Jul 28 20:19:53.502057 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.502037    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.505908 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.505861    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1e1ef\"/\"auth-test-model-a1e1ef-kserve-self-signed-certs\""
Jul 28 20:19:53.506061 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.506000    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1e1ef\"/\"default-dockercfg-k27f7\""
Jul 28 20:19:53.506061 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.506007    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a1e1ef\"/\"kube-root-ca.crt\""
Jul 28 20:19:53.506205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.506059    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a1e1ef\"/\"openshift-service-ca.crt\""
Jul 28 20:19:53.511305 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.511277    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkb6\" (UniqueName: \"kubernetes.io/projected/908b64fb-87fc-48be-9e58-d6484695cb70-kube-api-access-6gkb6\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.511439 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.511309    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-dshm\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.511439 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.511330    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-tmp-dir\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.511439 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.511363    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-model-cache\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.511613 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.511493    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908b64fb-87fc-48be-9e58-d6484695cb70-tls-certs\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.511613 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.511561    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-home\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.515554 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.515531    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"]
Jul 28 20:19:53.526614 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.526586    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:19:53.612783 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.612757    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkb6\" (UniqueName: \"kubernetes.io/projected/908b64fb-87fc-48be-9e58-d6484695cb70-kube-api-access-6gkb6\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.612883 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.612789    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-dshm\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.612883 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.612812    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-tmp-dir\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.612883 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.612831    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-model-cache\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.612883 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.612857    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908b64fb-87fc-48be-9e58-d6484695cb70-tls-certs\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.613142 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.612909    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-home\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.613264 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.613242    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-tmp-dir\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.613353 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.613309    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-model-cache\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.613420 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.613397    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-home\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.615319 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.615295    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-dshm\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.615586 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.615568    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908b64fb-87fc-48be-9e58-d6484695cb70-tls-certs\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.621349 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.621321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkb6\" (UniqueName: \"kubernetes.io/projected/908b64fb-87fc-48be-9e58-d6484695cb70-kube-api-access-6gkb6\") pod \"auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") " pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.812165 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.812138    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:19:53.944373 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:53.944352    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"]
Jul 28 20:19:53.946456 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:19:53.946421    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod908b64fb_87fc_48be_9e58_d6484695cb70.slice/crio-778e799d4257ff21f1eec49eef992c50b006f28dc24162d796d3cf98fa29530c WatchSource:0}: Error finding container 778e799d4257ff21f1eec49eef992c50b006f28dc24162d796d3cf98fa29530c: Status 404 returned error can't find the container with id 778e799d4257ff21f1eec49eef992c50b006f28dc24162d796d3cf98fa29530c
Jul 28 20:19:54.072743 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:54.072673    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf" event={"ID":"908b64fb-87fc-48be-9e58-d6484695cb70","Type":"ContainerStarted","Data":"9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e"}
Jul 28 20:19:54.072743 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:54.072711    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf" event={"ID":"908b64fb-87fc-48be-9e58-d6484695cb70","Type":"ContainerStarted","Data":"778e799d4257ff21f1eec49eef992c50b006f28dc24162d796d3cf98fa29530c"}
Jul 28 20:19:54.090808 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:54.090750    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf" podStartSLOduration=1.090732981 podStartE2EDuration="1.090732981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:54.090375329 +0000 UTC m=+2612.551133927" watchObservedRunningTime="2026-07-28 20:19:54.090732981 +0000 UTC m=+2612.551491583"
Jul 28 20:19:56.915359 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:19:56.915321    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:20:03.812641 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:03.812606    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:20:03.812641 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:03.812660    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:20:03.825229 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:03.825203    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:20:04.128358 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:04.128286    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:20:15.005544 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.005511    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"]
Jul 28 20:20:15.008821 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.008805    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.011524 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.011499    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"openshift-service-ca.crt\""
Jul 28 20:20:15.012625 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.012601    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"kube-root-ca.crt\""
Jul 28 20:20:15.012749 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.012604    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"auth-test-model-47f87c-kserve-self-signed-certs\""
Jul 28 20:20:15.012749 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.012680    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"default-dockercfg-5kpwc\""
Jul 28 20:20:15.022608 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.022590    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"]
Jul 28 20:20:15.085079 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.085054    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-dshm\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.085194 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.085097    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-model-cache\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.085194 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.085123    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063eb611-a2d0-4974-9682-76c4707d3231-tls-certs\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.085194 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.085182    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-home\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.085348 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.085201    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-tmp-dir\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.085348 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.085236    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxf5\" (UniqueName: \"kubernetes.io/projected/063eb611-a2d0-4974-9682-76c4707d3231-kube-api-access-fhxf5\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.095059 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.095030    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:20:15.186590 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.186565    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-home\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.186703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.186592    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-tmp-dir\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.186703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.186614    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxf5\" (UniqueName: \"kubernetes.io/projected/063eb611-a2d0-4974-9682-76c4707d3231-kube-api-access-fhxf5\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.186703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.186664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-dshm\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.186703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.186686    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-model-cache\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.186703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.186701    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063eb611-a2d0-4974-9682-76c4707d3231-tls-certs\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.187106 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.187082    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-tmp-dir\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.187184 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.187103    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-home\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.187184 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.187133    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-model-cache\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.189113 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.189093    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-dshm\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.189307 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.189293    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063eb611-a2d0-4974-9682-76c4707d3231-tls-certs\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.195234 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.195218    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxf5\" (UniqueName: \"kubernetes.io/projected/063eb611-a2d0-4974-9682-76c4707d3231-kube-api-access-fhxf5\") pod \"auth-test-model-47f87c-kserve-86f8dd85f-48vdr\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") " pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.318386 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.318358    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:15.448970 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:15.448936    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"]
Jul 28 20:20:15.451913 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:20:15.451865    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod063eb611_a2d0_4974_9682_76c4707d3231.slice/crio-f71cba9a31d7a3c4fa7c06dc6693f7b43fd50614ff9f2fb60eea6cf3ad7a9abb WatchSource:0}: Error finding container f71cba9a31d7a3c4fa7c06dc6693f7b43fd50614ff9f2fb60eea6cf3ad7a9abb: Status 404 returned error can't find the container with id f71cba9a31d7a3c4fa7c06dc6693f7b43fd50614ff9f2fb60eea6cf3ad7a9abb
Jul 28 20:20:16.171252 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:16.171214    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr" event={"ID":"063eb611-a2d0-4974-9682-76c4707d3231","Type":"ContainerStarted","Data":"ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9"}
Jul 28 20:20:16.171252 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:16.171256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr" event={"ID":"063eb611-a2d0-4974-9682-76c4707d3231","Type":"ContainerStarted","Data":"f71cba9a31d7a3c4fa7c06dc6693f7b43fd50614ff9f2fb60eea6cf3ad7a9abb"}
Jul 28 20:20:16.188241 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:16.188189    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr" podStartSLOduration=2.188172478 podStartE2EDuration="2.188172478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:20:16.187501372 +0000 UTC m=+2634.648259989" watchObservedRunningTime="2026-07-28 20:20:16.188172478 +0000 UTC m=+2634.648931078"
Jul 28 20:20:19.010494 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:19.010461    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:20:25.319156 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:25.319117    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:25.319156 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:25.319167    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:25.331610 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:25.331583    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:26.226932 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:26.226877    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:20:55.516313 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:20:55.516263    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:21:16.694499 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:21:16.694402    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:21:34.507240 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:21:34.507190    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:21:50.989563 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:21:50.989527    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:21:57.098853 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:21:57.098809    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:22:13.008831 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:22:13.008794    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:22:33.008472 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:22:33.008433    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:22:46.676167 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:22:46.676080    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:22:50.411912 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:22:50.411863    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:23:17.409763 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:23:17.409711    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:23:34.656858 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:23:34.656810    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:24:01.433880 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:01.433842    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:24:01.931953 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:01.931917    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:17.404937 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:17.404842    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:24:29.992039 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:29.992005    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:37.914752 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:37.914707    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:24:43.093882 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:43.093849    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"]
Jul 28 20:24:43.180145 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:43.180097    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:44.035866 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:44.035832    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"]
Jul 28 20:24:44.069353 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:44.069326    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:47.714952 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.714915    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"]
Jul 28 20:24:47.718104 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.718086    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.727594 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:24:47.727559    2555 status_manager.go:1018] "Failed to get status for pod" err="pods \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" is forbidden: User \"system:node:ip-10-0-138-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-a1e1ef\": no relationship found between node 'ip-10-0-138-173.ec2.internal' and this object" podUID="33c66b04-cf79-4d46-9599-6f059328f308" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.728032 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:24:47.728004    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"sub-test-model-a1e1ef-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-138-173.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-a1e1ef\": no relationship found between node 'ip-10-0-138-173.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-shared-a-a1e1ef\"/\"sub-test-model-a1e1ef-kserve-self-signed-certs\"" type="*v1.Secret"
Jul 28 20:24:47.765108 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.765085    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"]
Jul 28 20:24:47.766181 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.766156    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:24:47.780686 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.780660    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-home\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.780778 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.780697    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.780778 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.780737    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-tmp-dir\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.780852 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.780801    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-dshm\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.780919 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.780857    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-model-cache\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.780919 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.780876    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzcjm\" (UniqueName: \"kubernetes.io/projected/33c66b04-cf79-4d46-9599-6f059328f308-kube-api-access-zzcjm\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.881684 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.881646    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-tmp-dir\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.881800 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.881695    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-dshm\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.881800 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.881744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-model-cache\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.881800 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.881771    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzcjm\" (UniqueName: \"kubernetes.io/projected/33c66b04-cf79-4d46-9599-6f059328f308-kube-api-access-zzcjm\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.882014 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.881815    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-home\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.882014 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.881851    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.882097 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.882078    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-tmp-dir\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.882150 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.882132    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-model-cache\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.882208 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.882189    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-home\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.883864 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.883843    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-dshm\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:47.896072 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:47.896049    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzcjm\" (UniqueName: \"kubernetes.io/projected/33c66b04-cf79-4d46-9599-6f059328f308-kube-api-access-zzcjm\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:48.882564 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:24:48.882527    2555 secret.go:189] Couldn't get secret ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-self-signed-certs: failed to sync secret cache: timed out waiting for the condition
Jul 28 20:24:48.883062 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:24:48.882620    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs podName:33c66b04-cf79-4d46-9599-6f059328f308 nodeName:}" failed. No retries permitted until 2026-07-28 20:24:49.382600031 +0000 UTC m=+2907.843358611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs") pod "sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt" (UID: "33c66b04-cf79-4d46-9599-6f059328f308") : failed to sync secret cache: timed out waiting for the condition
Jul 28 20:24:49.258188 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:49.258122    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1e1ef\"/\"sub-test-model-a1e1ef-kserve-self-signed-certs\""
Jul 28 20:24:49.392151 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:49.392124    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:49.394669 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:49.394646    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs\") pod \"sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") " pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:49.527599 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:49.527535    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:49.669682 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:49.669646    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"]
Jul 28 20:24:49.674320 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:24:49.674297    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33c66b04_cf79_4d46_9599_6f059328f308.slice/crio-7622517e32ec3e8c0a26e6357e37253b3c85bf85a075000beaa430cddb3b7062 WatchSource:0}: Error finding container 7622517e32ec3e8c0a26e6357e37253b3c85bf85a075000beaa430cddb3b7062: Status 404 returned error can't find the container with id 7622517e32ec3e8c0a26e6357e37253b3c85bf85a075000beaa430cddb3b7062
Jul 28 20:24:50.401070 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:50.401026    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt" event={"ID":"33c66b04-cf79-4d46-9599-6f059328f308","Type":"ContainerStarted","Data":"ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7"}
Jul 28 20:24:50.401533 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:50.401078    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt" event={"ID":"33c66b04-cf79-4d46-9599-6f059328f308","Type":"ContainerStarted","Data":"7622517e32ec3e8c0a26e6357e37253b3c85bf85a075000beaa430cddb3b7062"}
Jul 28 20:24:55.246824 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:55.246761    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt" podStartSLOduration=8.246741162 podStartE2EDuration="8.246741162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:24:50.423811023 +0000 UTC m=+2908.884569622" watchObservedRunningTime="2026-07-28 20:24:55.246741162 +0000 UTC m=+2913.707499764"
Jul 28 20:24:55.247266 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:55.246928    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:24:58.112238 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.112194    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf" podUID="908b64fb-87fc-48be-9e58-d6484695cb70" containerName="main" containerID="cri-o://9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e" gracePeriod=45
Jul 28 20:24:58.364361 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.364307    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:24:58.443611 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.443577    2555 generic.go:356] "Generic (PLEG): container finished" podID="908b64fb-87fc-48be-9e58-d6484695cb70" containerID="9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e" exitCode=0
Jul 28 20:24:58.443760 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.443620    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf" event={"ID":"908b64fb-87fc-48be-9e58-d6484695cb70","Type":"ContainerDied","Data":"9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e"}
Jul 28 20:24:58.443760 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.443621    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"
Jul 28 20:24:58.443760 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.443645    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf" event={"ID":"908b64fb-87fc-48be-9e58-d6484695cb70","Type":"ContainerDied","Data":"778e799d4257ff21f1eec49eef992c50b006f28dc24162d796d3cf98fa29530c"}
Jul 28 20:24:58.443760 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.443661    2555 scope.go:117] "RemoveContainer" containerID="9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e"
Jul 28 20:24:58.451209 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.451193    2555 scope.go:117] "RemoveContainer" containerID="9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e"
Jul 28 20:24:58.451446 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:24:58.451430    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e\": container with ID starting with 9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e not found: ID does not exist" containerID="9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e"
Jul 28 20:24:58.451494 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.451456    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e"} err="failed to get container status \"9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e\": rpc error: code = NotFound desc = could not find container \"9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e\": container with ID starting with 9e6ba55d0ebc569b007375ecfd35cfc4e131808d83145bc0d34703051580201e not found: ID does not exist"
Jul 28 20:24:58.461806 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.461792    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-model-cache\") pod \"908b64fb-87fc-48be-9e58-d6484695cb70\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") "
Jul 28 20:24:58.461867 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.461818    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-dshm\") pod \"908b64fb-87fc-48be-9e58-d6484695cb70\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") "
Jul 28 20:24:58.461867 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.461857    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-tmp-dir\") pod \"908b64fb-87fc-48be-9e58-d6484695cb70\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") "
Jul 28 20:24:58.461975 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.461906    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gkb6\" (UniqueName: \"kubernetes.io/projected/908b64fb-87fc-48be-9e58-d6484695cb70-kube-api-access-6gkb6\") pod \"908b64fb-87fc-48be-9e58-d6484695cb70\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") "
Jul 28 20:24:58.461975 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.461952    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-home\") pod \"908b64fb-87fc-48be-9e58-d6484695cb70\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") "
Jul 28 20:24:58.462087 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.462028    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908b64fb-87fc-48be-9e58-d6484695cb70-tls-certs\") pod \"908b64fb-87fc-48be-9e58-d6484695cb70\" (UID: \"908b64fb-87fc-48be-9e58-d6484695cb70\") "
Jul 28 20:24:58.462142 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.462090    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-model-cache" (OuterVolumeSpecName: "model-cache") pod "908b64fb-87fc-48be-9e58-d6484695cb70" (UID: "908b64fb-87fc-48be-9e58-d6484695cb70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:58.462241 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.462213    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "908b64fb-87fc-48be-9e58-d6484695cb70" (UID: "908b64fb-87fc-48be-9e58-d6484695cb70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:58.462300 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.462239    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-home" (OuterVolumeSpecName: "home") pod "908b64fb-87fc-48be-9e58-d6484695cb70" (UID: "908b64fb-87fc-48be-9e58-d6484695cb70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:58.462364 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.462343    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:58.462364 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.462357    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:58.462440 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.462365    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:58.463866 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.463846    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-dshm" (OuterVolumeSpecName: "dshm") pod "908b64fb-87fc-48be-9e58-d6484695cb70" (UID: "908b64fb-87fc-48be-9e58-d6484695cb70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:58.464053 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.464029    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908b64fb-87fc-48be-9e58-d6484695cb70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "908b64fb-87fc-48be-9e58-d6484695cb70" (UID: "908b64fb-87fc-48be-9e58-d6484695cb70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:24:58.464119 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.464089    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908b64fb-87fc-48be-9e58-d6484695cb70-kube-api-access-6gkb6" (OuterVolumeSpecName: "kube-api-access-6gkb6") pod "908b64fb-87fc-48be-9e58-d6484695cb70" (UID: "908b64fb-87fc-48be-9e58-d6484695cb70"). InnerVolumeSpecName "kube-api-access-6gkb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:24:58.562801 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.562780    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908b64fb-87fc-48be-9e58-d6484695cb70-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:58.562801 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.562798    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908b64fb-87fc-48be-9e58-d6484695cb70-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:58.562945 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.562807    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gkb6\" (UniqueName: \"kubernetes.io/projected/908b64fb-87fc-48be-9e58-d6484695cb70-kube-api-access-6gkb6\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:58.768577 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.768550    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"]
Jul 28 20:24:58.777496 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:58.777471    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/auth-test-model-a1e1ef-kserve-68c9b857ff-flhvf"]
Jul 28 20:24:59.054659 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.054615    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr" podUID="063eb611-a2d0-4974-9682-76c4707d3231" containerName="main" containerID="cri-o://ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9" gracePeriod=45
Jul 28 20:24:59.294123 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.294099    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:24:59.371060 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371002    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-home\") pod \"063eb611-a2d0-4974-9682-76c4707d3231\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") "
Jul 28 20:24:59.371060 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371046    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhxf5\" (UniqueName: \"kubernetes.io/projected/063eb611-a2d0-4974-9682-76c4707d3231-kube-api-access-fhxf5\") pod \"063eb611-a2d0-4974-9682-76c4707d3231\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") "
Jul 28 20:24:59.371230 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371066    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-dshm\") pod \"063eb611-a2d0-4974-9682-76c4707d3231\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") "
Jul 28 20:24:59.371230 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371090    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-model-cache\") pod \"063eb611-a2d0-4974-9682-76c4707d3231\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") "
Jul 28 20:24:59.371230 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371195    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-tmp-dir\") pod \"063eb611-a2d0-4974-9682-76c4707d3231\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") "
Jul 28 20:24:59.371386 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371289    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063eb611-a2d0-4974-9682-76c4707d3231-tls-certs\") pod \"063eb611-a2d0-4974-9682-76c4707d3231\" (UID: \"063eb611-a2d0-4974-9682-76c4707d3231\") "
Jul 28 20:24:59.371386 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371305    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-home" (OuterVolumeSpecName: "home") pod "063eb611-a2d0-4974-9682-76c4707d3231" (UID: "063eb611-a2d0-4974-9682-76c4707d3231"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:59.371386 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371320    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-model-cache" (OuterVolumeSpecName: "model-cache") pod "063eb611-a2d0-4974-9682-76c4707d3231" (UID: "063eb611-a2d0-4974-9682-76c4707d3231"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:59.371514 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371425    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "063eb611-a2d0-4974-9682-76c4707d3231" (UID: "063eb611-a2d0-4974-9682-76c4707d3231"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:59.371628 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371612    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:59.371671 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371629    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:59.371671 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.371638    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:59.373246 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.373216    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063eb611-a2d0-4974-9682-76c4707d3231-kube-api-access-fhxf5" (OuterVolumeSpecName: "kube-api-access-fhxf5") pod "063eb611-a2d0-4974-9682-76c4707d3231" (UID: "063eb611-a2d0-4974-9682-76c4707d3231"). InnerVolumeSpecName "kube-api-access-fhxf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:24:59.373246 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.373238    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-dshm" (OuterVolumeSpecName: "dshm") pod "063eb611-a2d0-4974-9682-76c4707d3231" (UID: "063eb611-a2d0-4974-9682-76c4707d3231"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:24:59.373387 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.373266    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/063eb611-a2d0-4974-9682-76c4707d3231-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "063eb611-a2d0-4974-9682-76c4707d3231" (UID: "063eb611-a2d0-4974-9682-76c4707d3231"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:24:59.455182 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.455154    2555 generic.go:356] "Generic (PLEG): container finished" podID="063eb611-a2d0-4974-9682-76c4707d3231" containerID="ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9" exitCode=0
Jul 28 20:24:59.455289 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.455216    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"
Jul 28 20:24:59.455289 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.455240    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr" event={"ID":"063eb611-a2d0-4974-9682-76c4707d3231","Type":"ContainerDied","Data":"ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9"}
Jul 28 20:24:59.455411 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.455289    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr" event={"ID":"063eb611-a2d0-4974-9682-76c4707d3231","Type":"ContainerDied","Data":"f71cba9a31d7a3c4fa7c06dc6693f7b43fd50614ff9f2fb60eea6cf3ad7a9abb"}
Jul 28 20:24:59.455411 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.455311    2555 scope.go:117] "RemoveContainer" containerID="ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9"
Jul 28 20:24:59.463508 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.463490    2555 scope.go:117] "RemoveContainer" containerID="ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9"
Jul 28 20:24:59.463763 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:24:59.463745    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9\": container with ID starting with ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9 not found: ID does not exist" containerID="ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9"
Jul 28 20:24:59.463876 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.463830    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9"} err="failed to get container status \"ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9\": rpc error: code = NotFound desc = could not find container \"ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9\": container with ID starting with ba65b9b5586326bc9d0dc68303576cf1eb965e5c4f2318a876b97dbbfd36e1c9 not found: ID does not exist"
Jul 28 20:24:59.472113 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.472098    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/063eb611-a2d0-4974-9682-76c4707d3231-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:59.472177 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.472115    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxf5\" (UniqueName: \"kubernetes.io/projected/063eb611-a2d0-4974-9682-76c4707d3231-kube-api-access-fhxf5\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:59.472177 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.472125    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/063eb611-a2d0-4974-9682-76c4707d3231-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:24:59.488020 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.487997    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"]
Jul 28 20:24:59.493606 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.493587    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/auth-test-model-47f87c-kserve-86f8dd85f-48vdr"]
Jul 28 20:24:59.528228 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.528211    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:59.528285 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.528241    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:59.540598 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.540579    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:24:59.983477 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.983454    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="063eb611-a2d0-4974-9682-76c4707d3231" path="/var/lib/kubelet/pods/063eb611-a2d0-4974-9682-76c4707d3231/volumes"
Jul 28 20:24:59.983830 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:24:59.983816    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908b64fb-87fc-48be-9e58-d6484695cb70" path="/var/lib/kubelet/pods/908b64fb-87fc-48be-9e58-d6484695cb70/volumes"
Jul 28 20:25:00.472704 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:00.472678    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:25:30.033536 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.033499    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"]
Jul 28 20:25:30.034060 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.033954    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908b64fb-87fc-48be-9e58-d6484695cb70" containerName="main"
Jul 28 20:25:30.034060 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.033972    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="908b64fb-87fc-48be-9e58-d6484695cb70" containerName="main"
Jul 28 20:25:30.034060 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.033984    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="063eb611-a2d0-4974-9682-76c4707d3231" containerName="main"
Jul 28 20:25:30.034060 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.033992    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="063eb611-a2d0-4974-9682-76c4707d3231" containerName="main"
Jul 28 20:25:30.034276 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.034089    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="908b64fb-87fc-48be-9e58-d6484695cb70" containerName="main"
Jul 28 20:25:30.034276 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.034103    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="063eb611-a2d0-4974-9682-76c4707d3231" containerName="main"
Jul 28 20:25:30.038613 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.038589    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.041570 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.041545    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:25:30.042031 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.042014    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"default-dockercfg-5kpwc\""
Jul 28 20:25:30.043165 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.043141    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"openshift-service-ca.crt\""
Jul 28 20:25:30.043263 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.043141    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"sub-test-model-47f87c-kserve-self-signed-certs\""
Jul 28 20:25:30.043263 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.043224    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"kube-root-ca.crt\""
Jul 28 20:25:30.056630 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.056602    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"]
Jul 28 20:25:30.206187 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.206162    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-home\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.206302 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.206204    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4hwq\" (UniqueName: \"kubernetes.io/projected/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-kube-api-access-l4hwq\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.206302 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.206228    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-dshm\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.206390 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.206299    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tls-certs\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.206390 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.206343    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-model-cache\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.206390 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.206358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tmp-dir\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307052 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307026    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-model-cache\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307160 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307059    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tmp-dir\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307160 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-home\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307160 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307132    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4hwq\" (UniqueName: \"kubernetes.io/projected/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-kube-api-access-l4hwq\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307301 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307170    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-dshm\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307301 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307203    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tls-certs\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307487 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307467    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-home\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307554 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307474    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-model-cache\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.307554 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.307510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tmp-dir\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.309329 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.309301    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-dshm\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.309431 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.309416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tls-certs\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.319636 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.319609    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4hwq\" (UniqueName: \"kubernetes.io/projected/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-kube-api-access-l4hwq\") pod \"sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") " pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.350418 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.350401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:30.477187 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.477164    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"]
Jul 28 20:25:30.479181 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:25:30.479155    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ca2fd9a_3731_4ebb_9afe_84f9fbb08f6c.slice/crio-65cb92989b32131cb26b81564bc27f85d1934b9fe58d337884dc0663c7e5e5ba WatchSource:0}: Error finding container 65cb92989b32131cb26b81564bc27f85d1934b9fe58d337884dc0663c7e5e5ba: Status 404 returned error can't find the container with id 65cb92989b32131cb26b81564bc27f85d1934b9fe58d337884dc0663c7e5e5ba
Jul 28 20:25:30.595974 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.595884    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr" event={"ID":"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c","Type":"ContainerStarted","Data":"e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679"}
Jul 28 20:25:30.595974 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.595943    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr" event={"ID":"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c","Type":"ContainerStarted","Data":"65cb92989b32131cb26b81564bc27f85d1934b9fe58d337884dc0663c7e5e5ba"}
Jul 28 20:25:30.619506 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:30.619465    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr" podStartSLOduration=0.619450784 podStartE2EDuration="619.450784ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:25:30.61842807 +0000 UTC m=+2949.079186669" watchObservedRunningTime="2026-07-28 20:25:30.619450784 +0000 UTC m=+2949.080209383"
Jul 28 20:25:33.825933 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:33.825871    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:25:40.351532 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:40.351503    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:40.351986 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:40.351547    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:40.364009 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:40.363980    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:25:40.653767 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:25:40.653706    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:26:13.401969 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:13.401936    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:26:29.909128 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:29.909090    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:26:41.317075 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:41.317041    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"]
Jul 28 20:26:41.354298 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:41.354262    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:26:42.347823 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:42.347786    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"]
Jul 28 20:26:42.376980 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:42.376954    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:26:44.619785 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:44.619747    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:26:53.515929 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.515877    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"]
Jul 28 20:26:53.519235 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.519217    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.522559 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.522536    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1e1ef\"/\"rate-test-model-a1e1ef-kserve-self-signed-certs\""
Jul 28 20:26:53.530022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.530001    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"]
Jul 28 20:26:53.558358 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.558335    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:26:53.621262 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.621234    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2rqf\" (UniqueName: \"kubernetes.io/projected/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-kube-api-access-s2rqf\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.621372 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.621271    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tls-certs\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.621372 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.621297    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-home\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.621372 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.621355    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-model-cache\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.621497 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.621397    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tmp-dir\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.621497 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.621438    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-dshm\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.722631 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.722601    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2rqf\" (UniqueName: \"kubernetes.io/projected/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-kube-api-access-s2rqf\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.722736 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.722635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tls-certs\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.722736 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.722654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-home\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.722736 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.722705    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-model-cache\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.722736 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.722731    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tmp-dir\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.722923 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.722834    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-dshm\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.723232 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.723209    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tmp-dir\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.723297 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.723252    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-home\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.723297 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.723281    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-model-cache\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.725462 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.725440    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tls-certs\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.725536 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.725460    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-dshm\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.734428 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.734403    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2rqf\" (UniqueName: \"kubernetes.io/projected/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-kube-api-access-s2rqf\") pod \"rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") " pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.828498 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.828473    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:26:53.958189 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:53.958155    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"]
Jul 28 20:26:53.962962 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:26:53.962933    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbb2a71c_9e0b_42cf_b774_0670b4fbfa18.slice/crio-e453fa2c77f55b1ffb1756f9274aed12382d49fcc749367815e10e097f4723bc WatchSource:0}: Error finding container e453fa2c77f55b1ffb1756f9274aed12382d49fcc749367815e10e097f4723bc: Status 404 returned error can't find the container with id e453fa2c77f55b1ffb1756f9274aed12382d49fcc749367815e10e097f4723bc
Jul 28 20:26:54.970490 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:54.970456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w" event={"ID":"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18","Type":"ContainerStarted","Data":"79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4"}
Jul 28 20:26:54.970947 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:54.970502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w" event={"ID":"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18","Type":"ContainerStarted","Data":"e453fa2c77f55b1ffb1756f9274aed12382d49fcc749367815e10e097f4723bc"}
Jul 28 20:26:54.990856 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:54.990807    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w" podStartSLOduration=1.990792001 podStartE2EDuration="1.990792001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:26:54.98943748 +0000 UTC m=+3033.450196073" watchObservedRunningTime="2026-07-28 20:26:54.990792001 +0000 UTC m=+3033.451550599"
Jul 28 20:26:56.335987 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.335940    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt" podUID="33c66b04-cf79-4d46-9599-6f059328f308" containerName="main" containerID="cri-o://ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7" gracePeriod=45
Jul 28 20:26:56.593260 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.593203    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:26:56.744951 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.744918    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-home\") pod \"33c66b04-cf79-4d46-9599-6f059328f308\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") "
Jul 28 20:26:56.745092 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.744967    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzcjm\" (UniqueName: \"kubernetes.io/projected/33c66b04-cf79-4d46-9599-6f059328f308-kube-api-access-zzcjm\") pod \"33c66b04-cf79-4d46-9599-6f059328f308\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") "
Jul 28 20:26:56.745092 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.745008    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs\") pod \"33c66b04-cf79-4d46-9599-6f059328f308\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") "
Jul 28 20:26:56.745092 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.745037    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-tmp-dir\") pod \"33c66b04-cf79-4d46-9599-6f059328f308\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") "
Jul 28 20:26:56.745092 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.745060    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-dshm\") pod \"33c66b04-cf79-4d46-9599-6f059328f308\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") "
Jul 28 20:26:56.745092 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.745088    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-model-cache\") pod \"33c66b04-cf79-4d46-9599-6f059328f308\" (UID: \"33c66b04-cf79-4d46-9599-6f059328f308\") "
Jul 28 20:26:56.745345 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.745162    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-home" (OuterVolumeSpecName: "home") pod "33c66b04-cf79-4d46-9599-6f059328f308" (UID: "33c66b04-cf79-4d46-9599-6f059328f308"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:26:56.745399 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.745345    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33c66b04-cf79-4d46-9599-6f059328f308" (UID: "33c66b04-cf79-4d46-9599-6f059328f308"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:26:56.745399 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.745369    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-model-cache" (OuterVolumeSpecName: "model-cache") pod "33c66b04-cf79-4d46-9599-6f059328f308" (UID: "33c66b04-cf79-4d46-9599-6f059328f308"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:26:56.745399 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.745394    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:56.747244 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.747219    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c66b04-cf79-4d46-9599-6f059328f308-kube-api-access-zzcjm" (OuterVolumeSpecName: "kube-api-access-zzcjm") pod "33c66b04-cf79-4d46-9599-6f059328f308" (UID: "33c66b04-cf79-4d46-9599-6f059328f308"). InnerVolumeSpecName "kube-api-access-zzcjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:26:56.747347 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.747263    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33c66b04-cf79-4d46-9599-6f059328f308" (UID: "33c66b04-cf79-4d46-9599-6f059328f308"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:26:56.747347 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.747263    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-dshm" (OuterVolumeSpecName: "dshm") pod "33c66b04-cf79-4d46-9599-6f059328f308" (UID: "33c66b04-cf79-4d46-9599-6f059328f308"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:26:56.846672 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.846616    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzcjm\" (UniqueName: \"kubernetes.io/projected/33c66b04-cf79-4d46-9599-6f059328f308-kube-api-access-zzcjm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:56.846672 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.846638    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33c66b04-cf79-4d46-9599-6f059328f308-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:56.846672 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.846648    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:56.846672 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.846655    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:56.846672 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.846663    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33c66b04-cf79-4d46-9599-6f059328f308-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:56.981785 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.981755    2555 generic.go:356] "Generic (PLEG): container finished" podID="33c66b04-cf79-4d46-9599-6f059328f308" containerID="ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7" exitCode=0
Jul 28 20:26:56.981909 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.981801    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"
Jul 28 20:26:56.981909 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.981841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt" event={"ID":"33c66b04-cf79-4d46-9599-6f059328f308","Type":"ContainerDied","Data":"ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7"}
Jul 28 20:26:56.981909 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.981883    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt" event={"ID":"33c66b04-cf79-4d46-9599-6f059328f308","Type":"ContainerDied","Data":"7622517e32ec3e8c0a26e6357e37253b3c85bf85a075000beaa430cddb3b7062"}
Jul 28 20:26:56.982033 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.981919    2555 scope.go:117] "RemoveContainer" containerID="ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7"
Jul 28 20:26:56.990581 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.990567    2555 scope.go:117] "RemoveContainer" containerID="ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7"
Jul 28 20:26:56.990825 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:26:56.990806    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7\": container with ID starting with ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7 not found: ID does not exist" containerID="ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7"
Jul 28 20:26:56.990879 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:56.990833    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7"} err="failed to get container status \"ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7\": rpc error: code = NotFound desc = could not find container \"ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7\": container with ID starting with ae9a0a5fa3698f1f24ffad0ffb710ba7b77e735110da7af03ec5a1ea4132c5f7 not found: ID does not exist"
Jul 28 20:26:57.003701 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.003675    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"]
Jul 28 20:26:57.007787 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.007769    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/sub-test-model-a1e1ef-kserve-6788cf574f-nc2qt"]
Jul 28 20:26:57.366697 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.366664    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr" podUID="6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" containerName="main" containerID="cri-o://e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679" gracePeriod=45
Jul 28 20:26:57.610686 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.610663    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:26:57.753916 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.753841    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-dshm\") pod \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") "
Jul 28 20:26:57.753916 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.753880    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tmp-dir\") pod \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") "
Jul 28 20:26:57.754104 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.753923    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-home\") pod \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") "
Jul 28 20:26:57.754104 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.753964    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tls-certs\") pod \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") "
Jul 28 20:26:57.754104 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.753995    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4hwq\" (UniqueName: \"kubernetes.io/projected/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-kube-api-access-l4hwq\") pod \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") "
Jul 28 20:26:57.754104 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.754014    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-model-cache\") pod \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\" (UID: \"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c\") "
Jul 28 20:26:57.754302 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.754219    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" (UID: "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:26:57.754302 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.754236    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-home" (OuterVolumeSpecName: "home") pod "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" (UID: "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:26:57.754401 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.754379    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" (UID: "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:26:57.756062 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.756037    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-dshm" (OuterVolumeSpecName: "dshm") pod "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" (UID: "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:26:57.756170 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.756073    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" (UID: "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:26:57.756170 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.756124    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-kube-api-access-l4hwq" (OuterVolumeSpecName: "kube-api-access-l4hwq") pod "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" (UID: "6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c"). InnerVolumeSpecName "kube-api-access-l4hwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:26:57.854737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.854706    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:57.854737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.854738    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:57.854737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.854747    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:57.854939 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.854757    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4hwq\" (UniqueName: \"kubernetes.io/projected/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-kube-api-access-l4hwq\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:57.854939 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.854766    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:57.854939 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.854774    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:26:57.985537 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.985512    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33c66b04-cf79-4d46-9599-6f059328f308" path="/var/lib/kubelet/pods/33c66b04-cf79-4d46-9599-6f059328f308/volumes"
Jul 28 20:26:57.987917 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.987879    2555 generic.go:356] "Generic (PLEG): container finished" podID="6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" containerID="e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679" exitCode=0
Jul 28 20:26:57.988024 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.987935    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"
Jul 28 20:26:57.988024 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.987961    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr" event={"ID":"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c","Type":"ContainerDied","Data":"e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679"}
Jul 28 20:26:57.988024 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.987987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr" event={"ID":"6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c","Type":"ContainerDied","Data":"65cb92989b32131cb26b81564bc27f85d1934b9fe58d337884dc0663c7e5e5ba"}
Jul 28 20:26:57.988024 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.988003    2555 scope.go:117] "RemoveContainer" containerID="e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679"
Jul 28 20:26:57.996754 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.996739    2555 scope.go:117] "RemoveContainer" containerID="e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679"
Jul 28 20:26:57.997034 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:26:57.997015    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679\": container with ID starting with e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679 not found: ID does not exist" containerID="e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679"
Jul 28 20:26:57.997092 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:57.997043    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679"} err="failed to get container status \"e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679\": rpc error: code = NotFound desc = could not find container \"e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679\": container with ID starting with e129bb3107d1dad7f9034d70b62bafbf06950f6dc3dfb538d91e86af8a2dc679 not found: ID does not exist"
Jul 28 20:26:58.009802 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:58.009753    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"]
Jul 28 20:26:58.012547 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:58.012528    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/sub-test-model-47f87c-kserve-5ffc9c897c-qxwmr"]
Jul 28 20:26:58.709465 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:58.709431    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:26:59.984085 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:26:59.984052    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" path="/var/lib/kubelet/pods/6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c/volumes"
Jul 28 20:27:03.828781 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:03.828744    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:27:03.829271 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:03.828799    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:27:03.841495 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:03.841469    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:27:04.024091 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:04.024068    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:27:25.308698 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.308662    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"]
Jul 28 20:27:25.309243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.309012    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" containerName="main"
Jul 28 20:27:25.309243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.309027    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" containerName="main"
Jul 28 20:27:25.309243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.309044    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33c66b04-cf79-4d46-9599-6f059328f308" containerName="main"
Jul 28 20:27:25.309243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.309050    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="33c66b04-cf79-4d46-9599-6f059328f308" containerName="main"
Jul 28 20:27:25.309243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.309116    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca2fd9a-3731-4ebb-9afe-84f9fbb08f6c" containerName="main"
Jul 28 20:27:25.309243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.309122    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="33c66b04-cf79-4d46-9599-6f059328f308" containerName="main"
Jul 28 20:27:25.313378 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.313355    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.316255 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.316232    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"openshift-service-ca.crt\""
Jul 28 20:27:25.317259 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.317238    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"rate-test-model-47f87c-kserve-self-signed-certs\""
Jul 28 20:27:25.317516 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.317492    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"kube-root-ca.crt\""
Jul 28 20:27:25.317624 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.317521    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-47f87c\"/\"default-dockercfg-5kpwc\""
Jul 28 20:27:25.322149 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.322130    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"]
Jul 28 20:27:25.342638 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.342610    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:27:25.351053 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.351033    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbdmv\" (UniqueName: \"kubernetes.io/projected/e5ad3773-cc8e-4db6-b8b4-419c66801f32-kube-api-access-jbdmv\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.351138 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.351069    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-dshm\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.351138 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.351088    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tmp-dir\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.351208 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.351141    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tls-certs\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.351208 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.351174    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-home\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.351275 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.351240    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-model-cache\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452212 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452189    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-dshm\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452333 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tmp-dir\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452333 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tls-certs\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452333 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-home\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452333 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-model-cache\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452555 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452346    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbdmv\" (UniqueName: \"kubernetes.io/projected/e5ad3773-cc8e-4db6-b8b4-419c66801f32-kube-api-access-jbdmv\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452662 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452633    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tmp-dir\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452747 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452699    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-model-cache\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.452877 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.452840    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-home\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.454559 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.454531    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-dshm\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.454656 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.454642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tls-certs\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.461597 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.461575    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbdmv\" (UniqueName: \"kubernetes.io/projected/e5ad3773-cc8e-4db6-b8b4-419c66801f32-kube-api-access-jbdmv\") pod \"rate-test-model-47f87c-kserve-5bd498c7f-g2hwn\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") " pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.623141 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.623085    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:25.760614 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:25.760591    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"]
Jul 28 20:27:25.762223 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:27:25.762194    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5ad3773_cc8e_4db6_b8b4_419c66801f32.slice/crio-485ee3d6d5f97ae40a6cdb86ebb460800127bd4aa3ac924b9ae5548abfe79df5 WatchSource:0}: Error finding container 485ee3d6d5f97ae40a6cdb86ebb460800127bd4aa3ac924b9ae5548abfe79df5: Status 404 returned error can't find the container with id 485ee3d6d5f97ae40a6cdb86ebb460800127bd4aa3ac924b9ae5548abfe79df5
Jul 28 20:27:26.116115 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:26.116073    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn" event={"ID":"e5ad3773-cc8e-4db6-b8b4-419c66801f32","Type":"ContainerStarted","Data":"665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc"}
Jul 28 20:27:26.116115 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:26.116119    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn" event={"ID":"e5ad3773-cc8e-4db6-b8b4-419c66801f32","Type":"ContainerStarted","Data":"485ee3d6d5f97ae40a6cdb86ebb460800127bd4aa3ac924b9ae5548abfe79df5"}
Jul 28 20:27:26.135710 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:26.135660    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn" podStartSLOduration=1.135645998 podStartE2EDuration="1.135645998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:27:26.13484143 +0000 UTC m=+3064.595600031" watchObservedRunningTime="2026-07-28 20:27:26.135645998 +0000 UTC m=+3064.596404596"
Jul 28 20:27:29.302495 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:29.302459    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:27:35.623557 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:35.623515    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:35.623557 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:35.623570    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:35.636497 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:35.636469    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:27:36.049008 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:36.048976    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:27:36.172714 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:27:36.172689    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:28:00.914859 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:00.914818    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:28:21.201900 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:21.201841    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:28:27.597480 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:27.597440    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"]
Jul 28 20:28:27.622670 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:27.622635    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:28.523675 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:28.523638    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"]
Jul 28 20:28:28.560898 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:28.560851    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:39.404079 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:39.404038    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:28:42.616871 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.616818    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w" podUID="cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" containerName="main" containerID="cri-o://79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4" gracePeriod=45
Jul 28 20:28:42.869716 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.869661    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:28:42.916582 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.916555    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tls-certs\") pod \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") "
Jul 28 20:28:42.916712 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.916591    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-home\") pod \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") "
Jul 28 20:28:42.916712 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.916634    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tmp-dir\") pod \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") "
Jul 28 20:28:42.916712 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.916682    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2rqf\" (UniqueName: \"kubernetes.io/projected/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-kube-api-access-s2rqf\") pod \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") "
Jul 28 20:28:42.916946 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.916725    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-model-cache\") pod \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") "
Jul 28 20:28:42.916946 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.916761    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-dshm\") pod \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\" (UID: \"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18\") "
Jul 28 20:28:42.916946 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.916901    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-home" (OuterVolumeSpecName: "home") pod "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" (UID: "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:42.916946 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.916922    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" (UID: "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:42.917145 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.917027    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" (UID: "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:42.917145 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.917098    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:42.917145 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.917115    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:42.918541 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.918519    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" (UID: "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:28:42.918655 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.918637    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-kube-api-access-s2rqf" (OuterVolumeSpecName: "kube-api-access-s2rqf") pod "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" (UID: "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18"). InnerVolumeSpecName "kube-api-access-s2rqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:28:42.918719 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:42.918670    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-dshm" (OuterVolumeSpecName: "dshm") pod "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" (UID: "cbb2a71c-9e0b-42cf-b774-0670b4fbfa18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:43.018311 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.018289    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2rqf\" (UniqueName: \"kubernetes.io/projected/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-kube-api-access-s2rqf\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.018311 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.018310    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.018443 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.018321    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.018443 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.018329    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.468192 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.468157    2555 generic.go:356] "Generic (PLEG): container finished" podID="cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" containerID="79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4" exitCode=0
Jul 28 20:28:43.468354 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.468275    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w" event={"ID":"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18","Type":"ContainerDied","Data":"79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4"}
Jul 28 20:28:43.468354 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.468314    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w" event={"ID":"cbb2a71c-9e0b-42cf-b774-0670b4fbfa18","Type":"ContainerDied","Data":"e453fa2c77f55b1ffb1756f9274aed12382d49fcc749367815e10e097f4723bc"}
Jul 28 20:28:43.468354 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.468319    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"
Jul 28 20:28:43.468354 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.468335    2555 scope.go:117] "RemoveContainer" containerID="79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4"
Jul 28 20:28:43.477304 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.477286    2555 scope.go:117] "RemoveContainer" containerID="79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4"
Jul 28 20:28:43.477580 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:28:43.477557    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4\": container with ID starting with 79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4 not found: ID does not exist" containerID="79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4"
Jul 28 20:28:43.477688 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.477584    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4"} err="failed to get container status \"79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4\": rpc error: code = NotFound desc = could not find container \"79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4\": container with ID starting with 79a0cf9cd68d141c844b301783fc660cf380776277ac796523050dab5273fab4 not found: ID does not exist"
Jul 28 20:28:43.490557 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.490535    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"]
Jul 28 20:28:43.494703 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.494680    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a1e1ef/rate-test-model-a1e1ef-kserve-6d5cb9d695-2cs6w"]
Jul 28 20:28:43.541827 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.541799    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn" podUID="e5ad3773-cc8e-4db6-b8b4-419c66801f32" containerName="main" containerID="cri-o://665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc" gracePeriod=45
Jul 28 20:28:43.794166 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.794144    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:28:43.823315 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823290    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-home\") pod \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") "
Jul 28 20:28:43.823453 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823354    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbdmv\" (UniqueName: \"kubernetes.io/projected/e5ad3773-cc8e-4db6-b8b4-419c66801f32-kube-api-access-jbdmv\") pod \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") "
Jul 28 20:28:43.823453 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823371    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tls-certs\") pod \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") "
Jul 28 20:28:43.823453 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823392    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-dshm\") pod \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") "
Jul 28 20:28:43.823453 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823406    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-model-cache\") pod \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") "
Jul 28 20:28:43.823453 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823442    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tmp-dir\") pod \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\" (UID: \"e5ad3773-cc8e-4db6-b8b4-419c66801f32\") "
Jul 28 20:28:43.823751 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823536    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-home" (OuterVolumeSpecName: "home") pod "e5ad3773-cc8e-4db6-b8b4-419c66801f32" (UID: "e5ad3773-cc8e-4db6-b8b4-419c66801f32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:43.823751 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823691    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.823751 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823686    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5ad3773-cc8e-4db6-b8b4-419c66801f32" (UID: "e5ad3773-cc8e-4db6-b8b4-419c66801f32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:43.823751 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.823722    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5ad3773-cc8e-4db6-b8b4-419c66801f32" (UID: "e5ad3773-cc8e-4db6-b8b4-419c66801f32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:43.825390 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.825370    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-dshm" (OuterVolumeSpecName: "dshm") pod "e5ad3773-cc8e-4db6-b8b4-419c66801f32" (UID: "e5ad3773-cc8e-4db6-b8b4-419c66801f32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:28:43.825508 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.825423    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ad3773-cc8e-4db6-b8b4-419c66801f32-kube-api-access-jbdmv" (OuterVolumeSpecName: "kube-api-access-jbdmv") pod "e5ad3773-cc8e-4db6-b8b4-419c66801f32" (UID: "e5ad3773-cc8e-4db6-b8b4-419c66801f32"). InnerVolumeSpecName "kube-api-access-jbdmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:28:43.825508 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.825493    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5ad3773-cc8e-4db6-b8b4-419c66801f32" (UID: "e5ad3773-cc8e-4db6-b8b4-419c66801f32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:28:43.924676 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.924654    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbdmv\" (UniqueName: \"kubernetes.io/projected/e5ad3773-cc8e-4db6-b8b4-419c66801f32-kube-api-access-jbdmv\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.924676 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.924674    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.924804 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.924684    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.924804 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.924692    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.924804 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.924699    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5ad3773-cc8e-4db6-b8b4-419c66801f32-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:28:43.983845 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:43.983815    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" path="/var/lib/kubelet/pods/cbb2a71c-9e0b-42cf-b774-0670b4fbfa18/volumes"
Jul 28 20:28:44.475044 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.475007    2555 generic.go:356] "Generic (PLEG): container finished" podID="e5ad3773-cc8e-4db6-b8b4-419c66801f32" containerID="665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc" exitCode=0
Jul 28 20:28:44.475220 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.475066    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"
Jul 28 20:28:44.475220 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.475095    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn" event={"ID":"e5ad3773-cc8e-4db6-b8b4-419c66801f32","Type":"ContainerDied","Data":"665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc"}
Jul 28 20:28:44.475220 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.475140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn" event={"ID":"e5ad3773-cc8e-4db6-b8b4-419c66801f32","Type":"ContainerDied","Data":"485ee3d6d5f97ae40a6cdb86ebb460800127bd4aa3ac924b9ae5548abfe79df5"}
Jul 28 20:28:44.475220 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.475163    2555 scope.go:117] "RemoveContainer" containerID="665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc"
Jul 28 20:28:44.483561 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.483542    2555 scope.go:117] "RemoveContainer" containerID="665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc"
Jul 28 20:28:44.483874 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:28:44.483855    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc\": container with ID starting with 665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc not found: ID does not exist" containerID="665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc"
Jul 28 20:28:44.483979 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.483881    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc"} err="failed to get container status \"665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc\": rpc error: code = NotFound desc = could not find container \"665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc\": container with ID starting with 665b55b6dfd7271c2300cf45cc4742c78e9c5069d18ed412c986f7733099e0bc not found: ID does not exist"
Jul 28 20:28:44.512267 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.512237    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"]
Jul 28 20:28:44.516517 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:44.516497    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-47f87c/rate-test-model-47f87c-kserve-5bd498c7f-g2hwn"]
Jul 28 20:28:45.983761 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:45.983729    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ad3773-cc8e-4db6-b8b4-419c66801f32" path="/var/lib/kubelet/pods/e5ad3773-cc8e-4db6-b8b4-419c66801f32/volumes"
Jul 28 20:28:48.185226 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:48.185197    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:48.594968 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:48.594935    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:49.118636 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:49.118609    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:51.386598 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.386563    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:51.496470 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.496441    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"]
Jul 28 20:28:51.496940 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.496903    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" containerName="main"
Jul 28 20:28:51.496940 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.496923    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" containerName="main"
Jul 28 20:28:51.496940 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.496934    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ad3773-cc8e-4db6-b8b4-419c66801f32" containerName="main"
Jul 28 20:28:51.496940 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.496943    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ad3773-cc8e-4db6-b8b4-419c66801f32" containerName="main"
Jul 28 20:28:51.497179 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.497011    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ad3773-cc8e-4db6-b8b4-419c66801f32" containerName="main"
Jul 28 20:28:51.497179 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.497026    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb2a71c-9e0b-42cf-b774-0670b4fbfa18" containerName="main"
Jul 28 20:28:51.503115 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.502092    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:51.507085 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.507049    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:51.508215 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.508197    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-aa68dd-serving-cert\""
Jul 28 20:28:51.527539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.527520    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"]
Jul 28 20:28:51.578405 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.578376    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d65138-bff3-409f-bb7e-d91353fe6d6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:51.578502 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.578415    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:51.578502 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.578444    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz656\" (UniqueName: \"kubernetes.io/projected/90d65138-bff3-409f-bb7e-d91353fe6d6e-kube-api-access-zz656\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:51.607165 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.607142    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:51.679420 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.679369    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d65138-bff3-409f-bb7e-d91353fe6d6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:51.679420 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.679395    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:51.679420 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.679415    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz656\" (UniqueName: \"kubernetes.io/projected/90d65138-bff3-409f-bb7e-d91353fe6d6e-kube-api-access-zz656\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:51.679561 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:28:51.679522    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-serving-cert: secret "maas-api-e2e-ipp-a-aa68dd-serving-cert" not found
Jul 28 20:28:51.679604 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:28:51.679595    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls podName:90d65138-bff3-409f-bb7e-d91353fe6d6e nodeName:}" failed. No retries permitted until 2026-07-28 20:28:52.179572776 +0000 UTC m=+3150.640331357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls") pod "maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph" (UID: "90d65138-bff3-409f-bb7e-d91353fe6d6e") : secret "maas-api-e2e-ipp-a-aa68dd-serving-cert" not found
Jul 28 20:28:51.680138 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.680119    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d65138-bff3-409f-bb7e-d91353fe6d6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:51.691784 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:51.691761    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz656\" (UniqueName: \"kubernetes.io/projected/90d65138-bff3-409f-bb7e-d91353fe6d6e-kube-api-access-zz656\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:52.182527 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:52.182496    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:52.184876 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:52.184844    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:52.235193 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:52.235165    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:52.415111 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:52.415084    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:52.573649 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:52.573613    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"]
Jul 28 20:28:52.577189 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:28:52.577164    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d65138_bff3_409f_bb7e_d91353fe6d6e.slice/crio-7130c6749cc5d8f405d4e072bcdd83d0e3a032d6afc7b4d464edc4377269213b WatchSource:0}: Error finding container 7130c6749cc5d8f405d4e072bcdd83d0e3a032d6afc7b4d464edc4377269213b: Status 404 returned error can't find the container with id 7130c6749cc5d8f405d4e072bcdd83d0e3a032d6afc7b4d464edc4377269213b
Jul 28 20:28:52.578467 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:52.578449    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:28:52.606275 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:52.606251    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:52.770522 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:52.770448    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:28:53.520396 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:53.520360    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph" event={"ID":"90d65138-bff3-409f-bb7e-d91353fe6d6e","Type":"ContainerStarted","Data":"f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926"}
Jul 28 20:28:53.520757 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:53.520414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph" event={"ID":"90d65138-bff3-409f-bb7e-d91353fe6d6e","Type":"ContainerStarted","Data":"7130c6749cc5d8f405d4e072bcdd83d0e3a032d6afc7b4d464edc4377269213b"}
Jul 28 20:28:53.520757 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:53.520475    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:28:53.561408 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:53.561365    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph" podStartSLOduration=2.045829762 podStartE2EDuration="2.5613522s" totalImagesPullingTime="515.522438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:53.55845557 +0000 UTC m=+3152.019214169" watchObservedRunningTime="2026-07-28 20:28:53.5613522 +0000 UTC m=+3152.022110777"
Jul 28 20:28:59.465559 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:59.465514    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:28:59.531368 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:28:59.531342    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:29:16.332026 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:16.331990    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:17.000279 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:17.000247    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:17.836675 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:17.836645    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:18.032551 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:18.032518    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:20.590967 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.590931    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:20.692008 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.691983    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"]
Jul 28 20:29:20.695305 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.695285    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:20.702691 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.702673    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4ec840-serving-cert\""
Jul 28 20:29:20.731318 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.731298    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"]
Jul 28 20:29:20.783765 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.783741    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:20.794817 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.794795    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2bed5-1263-4182-ac0f-c4243973b3f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:20.794953 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.794835    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7wj\" (UniqueName: \"kubernetes.io/projected/f6b2bed5-1263-4182-ac0f-c4243973b3f5-kube-api-access-qs7wj\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:20.794953 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.794859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:20.895716 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.895656    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7wj\" (UniqueName: \"kubernetes.io/projected/f6b2bed5-1263-4182-ac0f-c4243973b3f5-kube-api-access-qs7wj\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:20.895716 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.895706    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:20.895872 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.895768    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2bed5-1263-4182-ac0f-c4243973b3f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:20.895964 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:29:20.895878    2555 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-serving-cert: secret "maas-api-e2e-ipp-b-4ec840-serving-cert" not found
Jul 28 20:29:20.896022 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:29:20.895982    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls podName:f6b2bed5-1263-4182-ac0f-c4243973b3f5 nodeName:}" failed. No retries permitted until 2026-07-28 20:29:21.395960905 +0000 UTC m=+3179.856719489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls") pod "maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df" (UID: "f6b2bed5-1263-4182-ac0f-c4243973b3f5") : secret "maas-api-e2e-ipp-b-4ec840-serving-cert" not found
Jul 28 20:29:20.896464 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.896445    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2bed5-1263-4182-ac0f-c4243973b3f5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:20.903493 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.903474    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:20.909185 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:20.909166    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7wj\" (UniqueName: \"kubernetes.io/projected/f6b2bed5-1263-4182-ac0f-c4243973b3f5-kube-api-access-qs7wj\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:21.361151 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.361125    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:21.400177 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.400153    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:21.402506 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.402490    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:21.495437 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.495415    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:21.596730 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.596707    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:21.604746 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.604730    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:21.647201 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.647174    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"]
Jul 28 20:29:21.650631 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.650606    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"]
Jul 28 20:29:21.650810 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.650791    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:21.654162 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.654142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:21.657277 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.657251    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4ec840\""
Jul 28 20:29:21.671705 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.671639    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"]
Jul 28 20:29:21.695726 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.695654    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"]
Jul 28 20:29:21.701776 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.701754    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qnj\" (UniqueName: \"kubernetes.io/projected/1ed43640-622a-4ab8-a73e-d75b2a85578e-kube-api-access-q7qnj\") pod \"payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj\" (UID: \"1ed43640-622a-4ab8-a73e-d75b2a85578e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:21.701937 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.701799    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmr4d\" (UniqueName: \"kubernetes.io/projected/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-kube-api-access-tmr4d\") pod \"payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877\" (UID: \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:21.701937 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.701831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed43640-622a-4ab8-a73e-d75b2a85578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj\" (UID: \"1ed43640-622a-4ab8-a73e-d75b2a85578e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:21.701937 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.701870    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877\" (UID: \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:21.758648 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.758623    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"]
Jul 28 20:29:21.762468 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:29:21.762445    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b2bed5_1263_4182_ac0f_c4243973b3f5.slice/crio-4c3349a092f5df7c2c3b3b0852df59dbd1f4efe7af4e9e7200a8d73a889cfae7 WatchSource:0}: Error finding container 4c3349a092f5df7c2c3b3b0852df59dbd1f4efe7af4e9e7200a8d73a889cfae7: Status 404 returned error can't find the container with id 4c3349a092f5df7c2c3b3b0852df59dbd1f4efe7af4e9e7200a8d73a889cfae7
Jul 28 20:29:21.803159 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.803136    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877\" (UID: \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:21.803257 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.803187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qnj\" (UniqueName: \"kubernetes.io/projected/1ed43640-622a-4ab8-a73e-d75b2a85578e-kube-api-access-q7qnj\") pod \"payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj\" (UID: \"1ed43640-622a-4ab8-a73e-d75b2a85578e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:21.803257 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.803226    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmr4d\" (UniqueName: \"kubernetes.io/projected/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-kube-api-access-tmr4d\") pod \"payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877\" (UID: \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:21.803257 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.803242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed43640-622a-4ab8-a73e-d75b2a85578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj\" (UID: \"1ed43640-622a-4ab8-a73e-d75b2a85578e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:21.803867 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.803849    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877\" (UID: \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:21.803936 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.803909    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed43640-622a-4ab8-a73e-d75b2a85578e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj\" (UID: \"1ed43640-622a-4ab8-a73e-d75b2a85578e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:21.821057 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.821038    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmr4d\" (UniqueName: \"kubernetes.io/projected/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-kube-api-access-tmr4d\") pod \"payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877\" (UID: \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:21.825031 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.825013    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qnj\" (UniqueName: \"kubernetes.io/projected/1ed43640-622a-4ab8-a73e-d75b2a85578e-kube-api-access-q7qnj\") pod \"payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj\" (UID: \"1ed43640-622a-4ab8-a73e-d75b2a85578e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:21.964514 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.964467    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:21.973292 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:21.973276    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:22.114639 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.114612    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"]
Jul 28 20:29:22.115871 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:29:22.115843    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed43640_622a_4ab8_a73e_d75b2a85578e.slice/crio-ea01d3feedb6f77122a5d576d52595013de61dcc6e4bf77ad813e48fd3dd9dbd WatchSource:0}: Error finding container ea01d3feedb6f77122a5d576d52595013de61dcc6e4bf77ad813e48fd3dd9dbd: Status 404 returned error can't find the container with id ea01d3feedb6f77122a5d576d52595013de61dcc6e4bf77ad813e48fd3dd9dbd
Jul 28 20:29:22.135707 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.135685    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"]
Jul 28 20:29:22.137761 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:29:22.137726    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8894fd8_7ade_4ab0_ba50_31685d7f6d8e.slice/crio-a8b415fc08233b742d27f9b7d1c9d3b1c1afeca8dd1bb8b11c132a233f08e543 WatchSource:0}: Error finding container a8b415fc08233b742d27f9b7d1c9d3b1c1afeca8dd1bb8b11c132a233f08e543: Status 404 returned error can't find the container with id a8b415fc08233b742d27f9b7d1c9d3b1c1afeca8dd1bb8b11c132a233f08e543
Jul 28 20:29:22.652985 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.652952    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df" event={"ID":"f6b2bed5-1263-4182-ac0f-c4243973b3f5","Type":"ContainerStarted","Data":"95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583"}
Jul 28 20:29:22.653428 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.653055    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df" event={"ID":"f6b2bed5-1263-4182-ac0f-c4243973b3f5","Type":"ContainerStarted","Data":"4c3349a092f5df7c2c3b3b0852df59dbd1f4efe7af4e9e7200a8d73a889cfae7"}
Jul 28 20:29:22.653428 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.653094    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:22.654539 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.654513    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj" event={"ID":"1ed43640-622a-4ab8-a73e-d75b2a85578e","Type":"ContainerStarted","Data":"c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3"}
Jul 28 20:29:22.654647 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.654559    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj" event={"ID":"1ed43640-622a-4ab8-a73e-d75b2a85578e","Type":"ContainerStarted","Data":"ea01d3feedb6f77122a5d576d52595013de61dcc6e4bf77ad813e48fd3dd9dbd"}
Jul 28 20:29:22.654647 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.654579    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:22.655987 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.655968    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877" event={"ID":"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e","Type":"ContainerStarted","Data":"7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0"}
Jul 28 20:29:22.656086 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.656006    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877" event={"ID":"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e","Type":"ContainerStarted","Data":"a8b415fc08233b742d27f9b7d1c9d3b1c1afeca8dd1bb8b11c132a233f08e543"}
Jul 28 20:29:22.656133 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.656104    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:22.697447 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.697405    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df" podStartSLOduration=2.275650388 podStartE2EDuration="2.697393165s" totalImagesPullingTime="421.742777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:22.695732638 +0000 UTC m=+3181.156491238" watchObservedRunningTime="2026-07-28 20:29:22.697393165 +0000 UTC m=+3181.158151764"
Jul 28 20:29:22.724679 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.724629    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877" podStartSLOduration=1.724613149 podStartE2EDuration="1.724613149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:22.723642855 +0000 UTC m=+3181.184401459" watchObservedRunningTime="2026-07-28 20:29:22.724613149 +0000 UTC m=+3181.185371752"
Jul 28 20:29:22.757830 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:22.757790    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj" podStartSLOduration=1.757774457 podStartE2EDuration="1.757774457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:29:22.756746438 +0000 UTC m=+3181.217505039" watchObservedRunningTime="2026-07-28 20:29:22.757774457 +0000 UTC m=+3181.218533058"
Jul 28 20:29:28.665903 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:28.665865    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:29:30.321981 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:30.321949    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:29:33.662459 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:33.662408    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:29:33.662966 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:33.662749    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:29:50.660207 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:50.660173    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:29:57.668901 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:29:57.668864    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877_e8894fd8-7ade-4ab0-ba50-31685d7f6d8e/payload-processing/0.log"
Jul 28 20:30:12.026269 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.026229    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"]
Jul 28 20:30:12.029433 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.029416    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.032705 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.032681    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-aa68dd\"/\"openshift-service-ca.crt\""
Jul 28 20:30:12.032838 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.032736    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-aa68dd\"/\"kube-root-ca.crt\""
Jul 28 20:30:12.033679 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.033659    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-aa68dd\"/\"default-dockercfg-g2lkw\""
Jul 28 20:30:12.033775 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.033731    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-aa68dd\"/\"ipp-route-aa68dd-kserve-self-signed-certs\""
Jul 28 20:30:12.056093 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.056066    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"]
Jul 28 20:30:12.115999 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.115974    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:12.173021 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.172997    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tmp-dir\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.173117 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.173024    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tls-certs\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.173117 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.173043    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-dshm\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.173117 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.173065    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-model-cache\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.173117 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.173084    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswjb\" (UniqueName: \"kubernetes.io/projected/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-kube-api-access-cswjb\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.173258 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.173166    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-home\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.273879 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.273856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-model-cache\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.273988 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.273925    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswjb\" (UniqueName: \"kubernetes.io/projected/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-kube-api-access-cswjb\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.273988 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.273963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-home\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.274084 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.274041    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tmp-dir\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.274084 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.274064    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tls-certs\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.274187 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.274093    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-dshm\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.274323 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.274302    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-model-cache\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.274394 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.274352    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-home\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.274436 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.274391    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tmp-dir\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.276368 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.276315    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-dshm\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.276512 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.276491    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tls-certs\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.283305 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.283281    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswjb\" (UniqueName: \"kubernetes.io/projected/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-kube-api-access-cswjb\") pod \"ipp-route-aa68dd-kserve-79b995dcbc-75gt7\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") " pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.338880 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.338847    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:12.472236 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.472210    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"]
Jul 28 20:30:12.473530 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:30:12.473505    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda39e285d_9f42_4f12_a7aa_4bf2bd7388dc.slice/crio-674c308eabb790b961fc4236c5dfefffd80d950beed57e57595d3ec9d840c404 WatchSource:0}: Error finding container 674c308eabb790b961fc4236c5dfefffd80d950beed57e57595d3ec9d840c404: Status 404 returned error can't find the container with id 674c308eabb790b961fc4236c5dfefffd80d950beed57e57595d3ec9d840c404
Jul 28 20:30:12.896838 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.896801    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7" event={"ID":"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc","Type":"ContainerStarted","Data":"aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40"}
Jul 28 20:30:12.897029 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.896855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7" event={"ID":"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc","Type":"ContainerStarted","Data":"674c308eabb790b961fc4236c5dfefffd80d950beed57e57595d3ec9d840c404"}
Jul 28 20:30:12.915696 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:12.915646    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7" podStartSLOduration=1.9156280940000001 podStartE2EDuration="1.915628094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:30:12.913628057 +0000 UTC m=+3231.374386658" watchObservedRunningTime="2026-07-28 20:30:12.915628094 +0000 UTC m=+3231.376386692"
Jul 28 20:30:15.909980 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:15.909863    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:30:22.339657 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:22.339623    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:22.340136 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:22.339685    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:22.352415 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:22.352384    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:22.954671 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:22.954642    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:30:43.318514 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:43.318482    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:30:55.110437 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:55.110409    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877_e8894fd8-7ade-4ab0-ba50-31685d7f6d8e/payload-processing/0.log"
Jul 28 20:30:56.218238 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:56.218203    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:56.317833 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:56.317802    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:58.397910 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:58.397866    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:58.534261 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:58.534234    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:58.645007 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:58.644980    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:59.243931 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.243905    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:59.682840 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.682785    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:59.782037 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.782013    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:30:59.877759 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.877735    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"]
Jul 28 20:30:59.881348 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.881329    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:30:59.886712 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.886691    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e5feff-dockercfg-zwfjh\""
Jul 28 20:30:59.892548 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.892530    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e5feff\""
Jul 28 20:30:59.913681 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.913659    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"]
Jul 28 20:30:59.936706 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.936641    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fafef7-0d56-4930-8d39-8e573a6a2a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk\" (UID: \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:30:59.936706 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:30:59.936687    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qdp\" (UniqueName: \"kubernetes.io/projected/e6fafef7-0d56-4930-8d39-8e573a6a2a84-kube-api-access-f7qdp\") pod \"payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk\" (UID: \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:00.037311 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:00.037286    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fafef7-0d56-4930-8d39-8e573a6a2a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk\" (UID: \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:00.037419 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:00.037319    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qdp\" (UniqueName: \"kubernetes.io/projected/e6fafef7-0d56-4930-8d39-8e573a6a2a84-kube-api-access-f7qdp\") pod \"payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk\" (UID: \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:00.037991 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:00.037972    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fafef7-0d56-4930-8d39-8e573a6a2a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk\" (UID: \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:00.056833 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:00.056808    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qdp\" (UniqueName: \"kubernetes.io/projected/e6fafef7-0d56-4930-8d39-8e573a6a2a84-kube-api-access-f7qdp\") pod \"payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk\" (UID: \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:00.191653 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:00.191601    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:00.328742 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:00.328703    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"]
Jul 28 20:31:00.329872 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:31:00.329840    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fafef7_0d56_4930_8d39_8e573a6a2a84.slice/crio-e59375b8751c3c699e7c00c7c567be47e00d7698f304ff014f2a6479de3fa6e9 WatchSource:0}: Error finding container e59375b8751c3c699e7c00c7c567be47e00d7698f304ff014f2a6479de3fa6e9: Status 404 returned error can't find the container with id e59375b8751c3c699e7c00c7c567be47e00d7698f304ff014f2a6479de3fa6e9
Jul 28 20:31:01.111309 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:01.111275    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk" event={"ID":"e6fafef7-0d56-4930-8d39-8e573a6a2a84","Type":"ContainerStarted","Data":"721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7"}
Jul 28 20:31:01.111708 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:01.111323    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk" event={"ID":"e6fafef7-0d56-4930-8d39-8e573a6a2a84","Type":"ContainerStarted","Data":"e59375b8751c3c699e7c00c7c567be47e00d7698f304ff014f2a6479de3fa6e9"}
Jul 28 20:31:01.111708 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:01.111444    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:01.139378 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:01.139323    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk" podStartSLOduration=2.139308782 podStartE2EDuration="2.139308782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:31:01.137417788 +0000 UTC m=+3279.598176402" watchObservedRunningTime="2026-07-28 20:31:01.139308782 +0000 UTC m=+3279.600067408"
Jul 28 20:31:03.029025 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:03.028994    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:31:12.118275 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:12.118230    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:23.369989 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:23.369924    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:29.252705 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.252673    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:29.289529 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.289503    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"]
Jul 28 20:31:29.289737 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.289694    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk" podUID="e6fafef7-0d56-4930-8d39-8e573a6a2a84" containerName="payload-pre-processing" containerID="cri-o://721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7" gracePeriod=30
Jul 28 20:31:29.359018 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.358992    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:29.460271 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.460247    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:29.540438 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.540411    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:29.579922 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.579878    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:29.660315 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.660286    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fafef7-0d56-4930-8d39-8e573a6a2a84-plugins-config-volume\") pod \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\" (UID: \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\") "
Jul 28 20:31:29.660427 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.660329    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7qdp\" (UniqueName: \"kubernetes.io/projected/e6fafef7-0d56-4930-8d39-8e573a6a2a84-kube-api-access-f7qdp\") pod \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\" (UID: \"e6fafef7-0d56-4930-8d39-8e573a6a2a84\") "
Jul 28 20:31:29.660716 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.660692    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fafef7-0d56-4930-8d39-8e573a6a2a84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6fafef7-0d56-4930-8d39-8e573a6a2a84" (UID: "e6fafef7-0d56-4930-8d39-8e573a6a2a84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:31:29.662354 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.662330    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fafef7-0d56-4930-8d39-8e573a6a2a84-kube-api-access-f7qdp" (OuterVolumeSpecName: "kube-api-access-f7qdp") pod "e6fafef7-0d56-4930-8d39-8e573a6a2a84" (UID: "e6fafef7-0d56-4930-8d39-8e573a6a2a84"). InnerVolumeSpecName "kube-api-access-f7qdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:31:29.761099 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.761076    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fafef7-0d56-4930-8d39-8e573a6a2a84-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:31:29.761099 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.761097    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qdp\" (UniqueName: \"kubernetes.io/projected/e6fafef7-0d56-4930-8d39-8e573a6a2a84-kube-api-access-f7qdp\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:31:29.963844 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:29.963814    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:30.244569 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.244502    2555 generic.go:356] "Generic (PLEG): container finished" podID="e6fafef7-0d56-4930-8d39-8e573a6a2a84" containerID="721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7" exitCode=0
Jul 28 20:31:30.244569 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.244550    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"
Jul 28 20:31:30.244569 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.244558    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk" event={"ID":"e6fafef7-0d56-4930-8d39-8e573a6a2a84","Type":"ContainerDied","Data":"721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7"}
Jul 28 20:31:30.244753 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.244602    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk" event={"ID":"e6fafef7-0d56-4930-8d39-8e573a6a2a84","Type":"ContainerDied","Data":"e59375b8751c3c699e7c00c7c567be47e00d7698f304ff014f2a6479de3fa6e9"}
Jul 28 20:31:30.244753 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.244623    2555 scope.go:117] "RemoveContainer" containerID="721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7"
Jul 28 20:31:30.259821 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.259796    2555 scope.go:117] "RemoveContainer" containerID="721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7"
Jul 28 20:31:30.260147 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:31:30.260126    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7\": container with ID starting with 721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7 not found: ID does not exist" containerID="721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7"
Jul 28 20:31:30.260193 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.260153    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7"} err="failed to get container status \"721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7\": rpc error: code = NotFound desc = could not find container \"721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7\": container with ID starting with 721b373defba671c0c68685c145f7352c731fcbecca23db7a4a1e463e38e7be7 not found: ID does not exist"
Jul 28 20:31:30.266217 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.266195    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"]
Jul 28 20:31:30.270067 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:30.270047    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e5feff-5486656b8d-fbdqk"]
Jul 28 20:31:31.985331 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:31.985279    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6fafef7-0d56-4930-8d39-8e573a6a2a84" path="/var/lib/kubelet/pods/e6fafef7-0d56-4930-8d39-8e573a6a2a84/volumes"
Jul 28 20:31:33.627624 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:33.627580    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:31:37.059629 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:37.059577    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:37.214127 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:37.214093    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:37.451272 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:37.451193    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:37.677032 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:37.677000    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:48.175080 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:48.175002    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:56.744934 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:56.744883    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:31:57.144079 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.144048    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"]
Jul 28 20:31:57.204261 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.204232    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:57.373504 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.373464    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph" podUID="90d65138-bff3-409f-bb7e-d91353fe6d6e" containerName="maas-api" containerID="cri-o://f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926" gracePeriod=30
Jul 28 20:31:57.619810 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.619789    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:31:57.664401 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.664378    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls\") pod \"90d65138-bff3-409f-bb7e-d91353fe6d6e\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") "
Jul 28 20:31:57.664530 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.664429    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d65138-bff3-409f-bb7e-d91353fe6d6e-openshift-service-ca\") pod \"90d65138-bff3-409f-bb7e-d91353fe6d6e\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") "
Jul 28 20:31:57.664595 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.664541    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz656\" (UniqueName: \"kubernetes.io/projected/90d65138-bff3-409f-bb7e-d91353fe6d6e-kube-api-access-zz656\") pod \"90d65138-bff3-409f-bb7e-d91353fe6d6e\" (UID: \"90d65138-bff3-409f-bb7e-d91353fe6d6e\") "
Jul 28 20:31:57.664820 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.664797    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d65138-bff3-409f-bb7e-d91353fe6d6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90d65138-bff3-409f-bb7e-d91353fe6d6e" (UID: "90d65138-bff3-409f-bb7e-d91353fe6d6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:31:57.666671 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.666648    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d65138-bff3-409f-bb7e-d91353fe6d6e-kube-api-access-zz656" (OuterVolumeSpecName: "kube-api-access-zz656") pod "90d65138-bff3-409f-bb7e-d91353fe6d6e" (UID: "90d65138-bff3-409f-bb7e-d91353fe6d6e"). InnerVolumeSpecName "kube-api-access-zz656". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:31:57.666793 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.666780    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90d65138-bff3-409f-bb7e-d91353fe6d6e" (UID: "90d65138-bff3-409f-bb7e-d91353fe6d6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:31:57.765498 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.765436    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz656\" (UniqueName: \"kubernetes.io/projected/90d65138-bff3-409f-bb7e-d91353fe6d6e-kube-api-access-zz656\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:31:57.765498 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.765461    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d65138-bff3-409f-bb7e-d91353fe6d6e-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:31:57.765498 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:57.765474    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d65138-bff3-409f-bb7e-d91353fe6d6e-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:31:58.379929 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.379874    2555 generic.go:356] "Generic (PLEG): container finished" podID="90d65138-bff3-409f-bb7e-d91353fe6d6e" containerID="f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926" exitCode=0
Jul 28 20:31:58.379929 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.379925    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph" event={"ID":"90d65138-bff3-409f-bb7e-d91353fe6d6e","Type":"ContainerDied","Data":"f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926"}
Jul 28 20:31:58.380144 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.379947    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"
Jul 28 20:31:58.380144 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.380014    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph" event={"ID":"90d65138-bff3-409f-bb7e-d91353fe6d6e","Type":"ContainerDied","Data":"7130c6749cc5d8f405d4e072bcdd83d0e3a032d6afc7b4d464edc4377269213b"}
Jul 28 20:31:58.380144 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.380045    2555 scope.go:117] "RemoveContainer" containerID="f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926"
Jul 28 20:31:58.388734 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.388715    2555 scope.go:117] "RemoveContainer" containerID="f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926"
Jul 28 20:31:58.389099 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:31:58.389077    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926\": container with ID starting with f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926 not found: ID does not exist" containerID="f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926"
Jul 28 20:31:58.389166 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.389106    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926"} err="failed to get container status \"f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926\": rpc error: code = NotFound desc = could not find container \"f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926\": container with ID starting with f4901787132e591a9245de25b44ae3653b42221c0c88dfca5e2515f40eeb6926 not found: ID does not exist"
Jul 28 20:31:58.405746 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.405720    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"]
Jul 28 20:31:58.409304 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.409279    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-aa68dd-6494d8c65f-6v7ph"]
Jul 28 20:31:58.848005 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:58.847975    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:31:59.983934 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:31:59.983883    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d65138-bff3-409f-bb7e-d91353fe6d6e" path="/var/lib/kubelet/pods/90d65138-bff3-409f-bb7e-d91353fe6d6e/volumes"
Jul 28 20:32:02.828817 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:02.828776    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"]
Jul 28 20:32:03.029883 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:03.029850    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:07.414402 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:07.414363    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:32:17.847188 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:17.847122    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7" podUID="a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" containerName="main" containerID="cri-o://aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40" gracePeriod=45
Jul 28 20:32:18.100324 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.100270    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:32:18.219070 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219037    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-dshm\") pod \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") "
Jul 28 20:32:18.219070 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219074    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-home\") pod \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") "
Jul 28 20:32:18.219272 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219116    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tmp-dir\") pod \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") "
Jul 28 20:32:18.219272 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219136    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tls-certs\") pod \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") "
Jul 28 20:32:18.219272 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219158    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-model-cache\") pod \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") "
Jul 28 20:32:18.219272 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219239    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cswjb\" (UniqueName: \"kubernetes.io/projected/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-kube-api-access-cswjb\") pod \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\" (UID: \"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc\") "
Jul 28 20:32:18.219515 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219407    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" (UID: "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:32:18.219515 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219425    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-home" (OuterVolumeSpecName: "home") pod "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" (UID: "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:32:18.219515 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.219493    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" (UID: "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:32:18.221188 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.221156    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-dshm" (OuterVolumeSpecName: "dshm") pod "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" (UID: "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:32:18.221324 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.221304    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" (UID: "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:32:18.221366 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.221309    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-kube-api-access-cswjb" (OuterVolumeSpecName: "kube-api-access-cswjb") pod "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" (UID: "a39e285d-9f42-4f12-a7aa-4bf2bd7388dc"). InnerVolumeSpecName "kube-api-access-cswjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:32:18.320362 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.320339    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswjb\" (UniqueName: \"kubernetes.io/projected/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-kube-api-access-cswjb\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:18.320362 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.320359    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-dshm\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:18.320489 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.320370    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-home\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:18.320489 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.320378    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tmp-dir\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:18.320489 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.320387    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-tls-certs\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:18.320489 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.320394    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc-model-cache\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:18.477030 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.476957    2555 generic.go:356] "Generic (PLEG): container finished" podID="a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" containerID="aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40" exitCode=0
Jul 28 20:32:18.477030 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.477016    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"
Jul 28 20:32:18.477182 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.477046    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7" event={"ID":"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc","Type":"ContainerDied","Data":"aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40"}
Jul 28 20:32:18.477182 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.477100    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7" event={"ID":"a39e285d-9f42-4f12-a7aa-4bf2bd7388dc","Type":"ContainerDied","Data":"674c308eabb790b961fc4236c5dfefffd80d950beed57e57595d3ec9d840c404"}
Jul 28 20:32:18.477182 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.477120    2555 scope.go:117] "RemoveContainer" containerID="aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40"
Jul 28 20:32:18.487141 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.487123    2555 scope.go:117] "RemoveContainer" containerID="aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40"
Jul 28 20:32:18.487464 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:32:18.487440    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40\": container with ID starting with aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40 not found: ID does not exist" containerID="aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40"
Jul 28 20:32:18.487517 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.487474    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40"} err="failed to get container status \"aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40\": rpc error: code = NotFound desc = could not find container \"aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40\": container with ID starting with aa1806225e2e6b0ee13935567948a82d3a4c76a9b3249f58085c86a53107eb40 not found: ID does not exist"
Jul 28 20:32:18.499638 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.499613    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"]
Jul 28 20:32:18.504172 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:18.504145    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-aa68dd/ipp-route-aa68dd-kserve-79b995dcbc-75gt7"]
Jul 28 20:32:19.985157 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:19.985117    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" path="/var/lib/kubelet/pods/a39e285d-9f42-4f12-a7aa-4bf2bd7388dc/volumes"
Jul 28 20:32:21.902615 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:21.902574    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:22.008198 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:22.008168    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:35.136294 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:35.136264    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:35.531031 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:35.530948    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:41.782380 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:41.782349    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:41.881691 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:41.881664    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:42.091245 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:42.091218    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:42.312069 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:42.312042    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:45.743998 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:45.743947    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:52.921766 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:52.921732    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"]
Jul 28 20:32:52.922202 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:52.922002    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df" podUID="f6b2bed5-1263-4182-ac0f-c4243973b3f5" containerName="maas-api" containerID="cri-o://95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583" gracePeriod=30
Jul 28 20:32:52.962487 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:52.962451    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:53.008656 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.008634    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"]
Jul 28 20:32:53.008853 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.008834    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877" podUID="e8894fd8-7ade-4ab0-ba50-31685d7f6d8e" containerName="payload-processing" containerID="cri-o://7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0" gracePeriod=30
Jul 28 20:32:53.028883 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.028855    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"]
Jul 28 20:32:53.029131 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.029108    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj" podUID="1ed43640-622a-4ab8-a73e-d75b2a85578e" containerName="payload-pre-processing" containerID="cri-o://c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3" gracePeriod=30
Jul 28 20:32:53.117393 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.117360    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:53.211540 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.211513    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:53.257516 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.257489    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:32:53.306541 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.306515    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:32:53.309297 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.309278    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:32:53.313520 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.313494    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:53.380207 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.380184    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls\") pod \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") "
Jul 28 20:32:53.380308 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.380219    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2bed5-1263-4182-ac0f-c4243973b3f5-openshift-service-ca\") pod \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") "
Jul 28 20:32:53.380308 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.380262    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs7wj\" (UniqueName: \"kubernetes.io/projected/f6b2bed5-1263-4182-ac0f-c4243973b3f5-kube-api-access-qs7wj\") pod \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\" (UID: \"f6b2bed5-1263-4182-ac0f-c4243973b3f5\") "
Jul 28 20:32:53.380564 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.380544    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b2bed5-1263-4182-ac0f-c4243973b3f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6b2bed5-1263-4182-ac0f-c4243973b3f5" (UID: "f6b2bed5-1263-4182-ac0f-c4243973b3f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:32:53.382239 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.382217    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b2bed5-1263-4182-ac0f-c4243973b3f5-kube-api-access-qs7wj" (OuterVolumeSpecName: "kube-api-access-qs7wj") pod "f6b2bed5-1263-4182-ac0f-c4243973b3f5" (UID: "f6b2bed5-1263-4182-ac0f-c4243973b3f5"). InnerVolumeSpecName "kube-api-access-qs7wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:32:53.382301 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.382256    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6b2bed5-1263-4182-ac0f-c4243973b3f5" (UID: "f6b2bed5-1263-4182-ac0f-c4243973b3f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:32:53.481549 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.481491    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-plugins-config-volume\") pod \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\" (UID: \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\") "
Jul 28 20:32:53.481549 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.481531    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmr4d\" (UniqueName: \"kubernetes.io/projected/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-kube-api-access-tmr4d\") pod \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\" (UID: \"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e\") "
Jul 28 20:32:53.481702 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.481566    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7qnj\" (UniqueName: \"kubernetes.io/projected/1ed43640-622a-4ab8-a73e-d75b2a85578e-kube-api-access-q7qnj\") pod \"1ed43640-622a-4ab8-a73e-d75b2a85578e\" (UID: \"1ed43640-622a-4ab8-a73e-d75b2a85578e\") "
Jul 28 20:32:53.481702 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.481595    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed43640-622a-4ab8-a73e-d75b2a85578e-plugins-config-volume\") pod \"1ed43640-622a-4ab8-a73e-d75b2a85578e\" (UID: \"1ed43640-622a-4ab8-a73e-d75b2a85578e\") "
Jul 28 20:32:53.481827 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.481807    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8894fd8-7ade-4ab0-ba50-31685d7f6d8e" (UID: "e8894fd8-7ade-4ab0-ba50-31685d7f6d8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:32:53.481879 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.481815    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6b2bed5-1263-4182-ac0f-c4243973b3f5-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:53.482032 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.482012    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6b2bed5-1263-4182-ac0f-c4243973b3f5-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:53.482074 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.482037    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed43640-622a-4ab8-a73e-d75b2a85578e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ed43640-622a-4ab8-a73e-d75b2a85578e" (UID: "1ed43640-622a-4ab8-a73e-d75b2a85578e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:32:53.482074 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.482042    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs7wj\" (UniqueName: \"kubernetes.io/projected/f6b2bed5-1263-4182-ac0f-c4243973b3f5-kube-api-access-qs7wj\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:53.483389 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.483367    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-kube-api-access-tmr4d" (OuterVolumeSpecName: "kube-api-access-tmr4d") pod "e8894fd8-7ade-4ab0-ba50-31685d7f6d8e" (UID: "e8894fd8-7ade-4ab0-ba50-31685d7f6d8e"). InnerVolumeSpecName "kube-api-access-tmr4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:32:53.483455 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.483439    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed43640-622a-4ab8-a73e-d75b2a85578e-kube-api-access-q7qnj" (OuterVolumeSpecName: "kube-api-access-q7qnj") pod "1ed43640-622a-4ab8-a73e-d75b2a85578e" (UID: "1ed43640-622a-4ab8-a73e-d75b2a85578e"). InnerVolumeSpecName "kube-api-access-q7qnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:32:53.582537 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.582515    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:53.582537 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.582536    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmr4d\" (UniqueName: \"kubernetes.io/projected/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e-kube-api-access-tmr4d\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:53.582648 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.582545    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7qnj\" (UniqueName: \"kubernetes.io/projected/1ed43640-622a-4ab8-a73e-d75b2a85578e-kube-api-access-q7qnj\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:53.582648 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.582553    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ed43640-622a-4ab8-a73e-d75b2a85578e-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:32:53.630380 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.630353    2555 generic.go:356] "Generic (PLEG): container finished" podID="f6b2bed5-1263-4182-ac0f-c4243973b3f5" containerID="95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583" exitCode=0
Jul 28 20:32:53.630469 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.630396    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"
Jul 28 20:32:53.630469 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.630436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df" event={"ID":"f6b2bed5-1263-4182-ac0f-c4243973b3f5","Type":"ContainerDied","Data":"95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583"}
Jul 28 20:32:53.630581 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.630488    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df" event={"ID":"f6b2bed5-1263-4182-ac0f-c4243973b3f5","Type":"ContainerDied","Data":"4c3349a092f5df7c2c3b3b0852df59dbd1f4efe7af4e9e7200a8d73a889cfae7"}
Jul 28 20:32:53.630581 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.630518    2555 scope.go:117] "RemoveContainer" containerID="95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583"
Jul 28 20:32:53.631805 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.631780    2555 generic.go:356] "Generic (PLEG): container finished" podID="1ed43640-622a-4ab8-a73e-d75b2a85578e" containerID="c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3" exitCode=0
Jul 28 20:32:53.631924 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.631859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj" event={"ID":"1ed43640-622a-4ab8-a73e-d75b2a85578e","Type":"ContainerDied","Data":"c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3"}
Jul 28 20:32:53.631924 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.631901    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"
Jul 28 20:32:53.632040 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.631954    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj" event={"ID":"1ed43640-622a-4ab8-a73e-d75b2a85578e","Type":"ContainerDied","Data":"ea01d3feedb6f77122a5d576d52595013de61dcc6e4bf77ad813e48fd3dd9dbd"}
Jul 28 20:32:53.633386 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.633366    2555 generic.go:356] "Generic (PLEG): container finished" podID="e8894fd8-7ade-4ab0-ba50-31685d7f6d8e" containerID="7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0" exitCode=0
Jul 28 20:32:53.633507 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.633395    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877" event={"ID":"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e","Type":"ContainerDied","Data":"7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0"}
Jul 28 20:32:53.633507 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.633419    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"
Jul 28 20:32:53.633507 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.633455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877" event={"ID":"e8894fd8-7ade-4ab0-ba50-31685d7f6d8e","Type":"ContainerDied","Data":"a8b415fc08233b742d27f9b7d1c9d3b1c1afeca8dd1bb8b11c132a233f08e543"}
Jul 28 20:32:53.639400 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.639371    2555 scope.go:117] "RemoveContainer" containerID="95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583"
Jul 28 20:32:53.639811 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:32:53.639705    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583\": container with ID starting with 95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583 not found: ID does not exist" containerID="95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583"
Jul 28 20:32:53.639811 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.639744    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583"} err="failed to get container status \"95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583\": rpc error: code = NotFound desc = could not find container \"95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583\": container with ID starting with 95f712e8fd824e1827f46ad0cb32c2fd2103c47258dcbb65d0f788e043cf0583 not found: ID does not exist"
Jul 28 20:32:53.639811 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.639769    2555 scope.go:117] "RemoveContainer" containerID="c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3"
Jul 28 20:32:53.648391 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.648372    2555 scope.go:117] "RemoveContainer" containerID="c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3"
Jul 28 20:32:53.648618 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:32:53.648599    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3\": container with ID starting with c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3 not found: ID does not exist" containerID="c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3"
Jul 28 20:32:53.648674 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.648623    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3"} err="failed to get container status \"c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3\": rpc error: code = NotFound desc = could not find container \"c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3\": container with ID starting with c40cd6c6bed52c959e4d254589c15dbffa90a1b3e52880bc337a84829b0199a3 not found: ID does not exist"
Jul 28 20:32:53.648674 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.648637    2555 scope.go:117] "RemoveContainer" containerID="7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0"
Jul 28 20:32:53.657532 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.657513    2555 scope.go:117] "RemoveContainer" containerID="7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0"
Jul 28 20:32:53.657820 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:32:53.657796    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0\": container with ID starting with 7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0 not found: ID does not exist" containerID="7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0"
Jul 28 20:32:53.657955 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.657830    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0"} err="failed to get container status \"7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0\": rpc error: code = NotFound desc = could not find container \"7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0\": container with ID starting with 7a4623044ba1c23d48dad38e6ebac706acf27e016f92db36bf9a91a3e9a385d0 not found: ID does not exist"
Jul 28 20:32:53.664649 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.664627    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"]
Jul 28 20:32:53.672527 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.672508    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4ec840-7cdc5d8568-tb6df"]
Jul 28 20:32:53.685014 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.684992    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"]
Jul 28 20:32:53.691758 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.691741    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4ec840-7989d44c48-hh877"]
Jul 28 20:32:53.695411 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.695391    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:53.708739 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.708720    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"]
Jul 28 20:32:53.715227 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.715210    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4ec840-7dd797dd84-ldxvj"]
Jul 28 20:32:53.983546 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.983519    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ed43640-622a-4ab8-a73e-d75b2a85578e" path="/var/lib/kubelet/pods/1ed43640-622a-4ab8-a73e-d75b2a85578e/volumes"
Jul 28 20:32:53.983870 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.983855    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8894fd8-7ade-4ab0-ba50-31685d7f6d8e" path="/var/lib/kubelet/pods/e8894fd8-7ade-4ab0-ba50-31685d7f6d8e/volumes"
Jul 28 20:32:53.984226 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:53.984212    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b2bed5-1263-4182-ac0f-c4243973b3f5" path="/var/lib/kubelet/pods/f6b2bed5-1263-4182-ac0f-c4243973b3f5/volumes"
Jul 28 20:32:56.112440 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:56.112400    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:32:59.871069 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:59.871030    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:32:59.971989 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:32:59.971951    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:00.113122 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:00.113092    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:00.314845 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:00.314813    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:02.719967 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:02.719926    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:06.620612 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:06.620568    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:11.129100 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.129060    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"]
Jul 28 20:33:11.129594 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.129375    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk" podUID="d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" containerName="maas-api" containerID="cri-o://fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23" gracePeriod=30
Jul 28 20:33:11.192236 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.192203    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"]
Jul 28 20:33:11.192719 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.192491    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv" podUID="9b4fccce-07eb-4608-9704-bc4fd3071325" containerName="payload-processing" containerID="cri-o://33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96" gracePeriod=30
Jul 28 20:33:11.430826 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.430806    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:33:11.451096 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.451076    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:33:11.488986 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.488964    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:11.510849 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.510822    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4fccce-07eb-4608-9704-bc4fd3071325-plugins-config-volume\") pod \"9b4fccce-07eb-4608-9704-bc4fd3071325\" (UID: \"9b4fccce-07eb-4608-9704-bc4fd3071325\") "
Jul 28 20:33:11.510950 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.510867    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgk85\" (UniqueName: \"kubernetes.io/projected/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-kube-api-access-wgk85\") pod \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") "
Jul 28 20:33:11.510950 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.510927    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls\") pod \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") "
Jul 28 20:33:11.511421 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.510955    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n2gt\" (UniqueName: \"kubernetes.io/projected/9b4fccce-07eb-4608-9704-bc4fd3071325-kube-api-access-8n2gt\") pod \"9b4fccce-07eb-4608-9704-bc4fd3071325\" (UID: \"9b4fccce-07eb-4608-9704-bc4fd3071325\") "
Jul 28 20:33:11.511421 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.511125    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-openshift-service-ca\") pod \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\" (UID: \"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9\") "
Jul 28 20:33:11.511692 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.511232    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4fccce-07eb-4608-9704-bc4fd3071325-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b4fccce-07eb-4608-9704-bc4fd3071325" (UID: "9b4fccce-07eb-4608-9704-bc4fd3071325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:33:11.511791 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.511720    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" (UID: "d2bc1188-8ae2-423b-8c88-fb7fc1759eb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:33:11.511867 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.511829    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:33:11.511867 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.511848    2555 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4fccce-07eb-4608-9704-bc4fd3071325-plugins-config-volume\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:33:11.513414 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.513369    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-kube-api-access-wgk85" (OuterVolumeSpecName: "kube-api-access-wgk85") pod "d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" (UID: "d2bc1188-8ae2-423b-8c88-fb7fc1759eb9"). InnerVolumeSpecName "kube-api-access-wgk85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:33:11.513645 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.513618    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4fccce-07eb-4608-9704-bc4fd3071325-kube-api-access-8n2gt" (OuterVolumeSpecName: "kube-api-access-8n2gt") pod "9b4fccce-07eb-4608-9704-bc4fd3071325" (UID: "9b4fccce-07eb-4608-9704-bc4fd3071325"). InnerVolumeSpecName "kube-api-access-8n2gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:33:11.513735 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.513675    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" (UID: "d2bc1188-8ae2-423b-8c88-fb7fc1759eb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:33:11.612523 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.612495    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgk85\" (UniqueName: \"kubernetes.io/projected/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-kube-api-access-wgk85\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:33:11.612523 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.612518    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:33:11.612682 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.612531    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n2gt\" (UniqueName: \"kubernetes.io/projected/9b4fccce-07eb-4608-9704-bc4fd3071325-kube-api-access-8n2gt\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:33:11.613304 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.613284    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:11.716741 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.716679    2555 generic.go:356] "Generic (PLEG): container finished" podID="9b4fccce-07eb-4608-9704-bc4fd3071325" containerID="33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96" exitCode=0
Jul 28 20:33:11.716741 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.716723    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"
Jul 28 20:33:11.716912 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.716764    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv" event={"ID":"9b4fccce-07eb-4608-9704-bc4fd3071325","Type":"ContainerDied","Data":"33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96"}
Jul 28 20:33:11.716912 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.716820    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv" event={"ID":"9b4fccce-07eb-4608-9704-bc4fd3071325","Type":"ContainerDied","Data":"13951e0fd8eefd229091105634c49f61d47a7c43a52ae367433e0eaf5dbe1d61"}
Jul 28 20:33:11.716912 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.716849    2555 scope.go:117] "RemoveContainer" containerID="33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96"
Jul 28 20:33:11.718084 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.718065    2555 generic.go:356] "Generic (PLEG): container finished" podID="d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" containerID="fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23" exitCode=0
Jul 28 20:33:11.718146 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.718103    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk" event={"ID":"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9","Type":"ContainerDied","Data":"fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23"}
Jul 28 20:33:11.718146 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.718109    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"
Jul 28 20:33:11.718269 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.718139    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk" event={"ID":"d2bc1188-8ae2-423b-8c88-fb7fc1759eb9","Type":"ContainerDied","Data":"cb32c9e2ff59de172ccf174e08e8666de9938bf3a4213da5c827cdb16719aafa"}
Jul 28 20:33:11.725981 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.725961    2555 scope.go:117] "RemoveContainer" containerID="33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96"
Jul 28 20:33:11.726537 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:33:11.726264    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96\": container with ID starting with 33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96 not found: ID does not exist" containerID="33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96"
Jul 28 20:33:11.726707 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.726668    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96"} err="failed to get container status \"33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96\": rpc error: code = NotFound desc = could not find container \"33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96\": container with ID starting with 33f019d0f0f2924454dd8c58512bc77970fe510d53f4bc07ecb955e730c59c96 not found: ID does not exist"
Jul 28 20:33:11.726707 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.726701    2555 scope.go:117] "RemoveContainer" containerID="fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23"
Jul 28 20:33:11.739768 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.739747    2555 scope.go:117] "RemoveContainer" containerID="fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23"
Jul 28 20:33:11.740022 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:33:11.740004    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23\": container with ID starting with fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23 not found: ID does not exist" containerID="fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23"
Jul 28 20:33:11.740071 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.740029    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23"} err="failed to get container status \"fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23\": rpc error: code = NotFound desc = could not find container \"fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23\": container with ID starting with fad37e6f5d10919ef088b74f5f8869c0da84f9d0251198fd274a1c23bc4d4f23 not found: ID does not exist"
Jul 28 20:33:11.747534 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.747513    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"]
Jul 28 20:33:11.751403 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.751381    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1e1ef-7bf69ff8d4-jx4zk"]
Jul 28 20:33:11.764477 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.764451    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"]
Jul 28 20:33:11.768264 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.768245    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-a1e1ef-6c9db4c698-9drhv"]
Jul 28 20:33:11.872247 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.872225    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:11.984358 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.984307    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b4fccce-07eb-4608-9704-bc4fd3071325" path="/var/lib/kubelet/pods/9b4fccce-07eb-4608-9704-bc4fd3071325/volumes"
Jul 28 20:33:11.984670 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:11.984657    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" path="/var/lib/kubelet/pods/d2bc1188-8ae2-423b-8c88-fb7fc1759eb9/volumes"
Jul 28 20:33:14.114686 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:14.114612    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:33:18.052424 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:18.052390    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:18.119832 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:18.119803    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:18.302938 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:18.302840    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:18.527319 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:18.527278    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:29.175854 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.175813    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"]
Jul 28 20:33:29.176326 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.176103    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh" podUID="8eafb2d8-b01e-4a4b-b407-a6659afe8f31" containerName="maas-api" containerID="cri-o://b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2" gracePeriod=30
Jul 28 20:33:29.208882 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.208857    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:29.285057 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.285030    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:29.353817 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.353786    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:29.417092 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.417072    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:33:29.422185 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.422166    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:29.534205 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.534146    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-openshift-service-ca\") pod \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") "
Jul 28 20:33:29.534333 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.534215    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsn7b\" (UniqueName: \"kubernetes.io/projected/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-kube-api-access-tsn7b\") pod \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") "
Jul 28 20:33:29.534333 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.534238    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls\") pod \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\" (UID: \"8eafb2d8-b01e-4a4b-b407-a6659afe8f31\") "
Jul 28 20:33:29.534556 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.534526    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8eafb2d8-b01e-4a4b-b407-a6659afe8f31" (UID: "8eafb2d8-b01e-4a4b-b407-a6659afe8f31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:33:29.536276 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.536252    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-kube-api-access-tsn7b" (OuterVolumeSpecName: "kube-api-access-tsn7b") pod "8eafb2d8-b01e-4a4b-b407-a6659afe8f31" (UID: "8eafb2d8-b01e-4a4b-b407-a6659afe8f31"). InnerVolumeSpecName "kube-api-access-tsn7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:33:29.536364 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.536347    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8eafb2d8-b01e-4a4b-b407-a6659afe8f31" (UID: "8eafb2d8-b01e-4a4b-b407-a6659afe8f31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:33:29.634799 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.634777    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsn7b\" (UniqueName: \"kubernetes.io/projected/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-kube-api-access-tsn7b\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:33:29.634799 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.634797    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-maas-api-tls\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:33:29.634948 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.634806    2555 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8eafb2d8-b01e-4a4b-b407-a6659afe8f31-openshift-service-ca\") on node \"ip-10-0-138-173.ec2.internal\" DevicePath \"\""
Jul 28 20:33:29.800327 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.800301    2555 generic.go:356] "Generic (PLEG): container finished" podID="8eafb2d8-b01e-4a4b-b407-a6659afe8f31" containerID="b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2" exitCode=0
Jul 28 20:33:29.800498 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.800346    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"
Jul 28 20:33:29.800498 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.800369    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh" event={"ID":"8eafb2d8-b01e-4a4b-b407-a6659afe8f31","Type":"ContainerDied","Data":"b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2"}
Jul 28 20:33:29.800498 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.800411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh" event={"ID":"8eafb2d8-b01e-4a4b-b407-a6659afe8f31","Type":"ContainerDied","Data":"4b8e7c4583b6be8958683edbbb2a5e7cde1caa2ae86dff7d266e2d96c0b15dab"}
Jul 28 20:33:29.800498 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.800433    2555 scope.go:117] "RemoveContainer" containerID="b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2"
Jul 28 20:33:29.808873 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.808857    2555 scope.go:117] "RemoveContainer" containerID="b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2"
Jul 28 20:33:29.809166 ip-10-0-138-173 kubenswrapper[2555]: E0728 20:33:29.809145    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2\": container with ID starting with b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2 not found: ID does not exist" containerID="b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2"
Jul 28 20:33:29.809236 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.809180    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2"} err="failed to get container status \"b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2\": rpc error: code = NotFound desc = could not find container \"b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2\": container with ID starting with b0870e92a95c50ee7ed23436a7f5739e0bfb1a9703f45418d6227b45e388b2a2 not found: ID does not exist"
Jul 28 20:33:29.822714 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.822692    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"]
Jul 28 20:33:29.827222 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.827201    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-47f87c-867c4b947c-mqfnh"]
Jul 28 20:33:29.962546 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.962522    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:29.988341 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:29.984540    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8eafb2d8-b01e-4a4b-b407-a6659afe8f31" path="/var/lib/kubelet/pods/8eafb2d8-b01e-4a4b-b407-a6659afe8f31/volumes"
Jul 28 20:33:32.015154 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:32.015112    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lwx8z"]
Jul 28 20:33:36.040088 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:36.040056    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:36.140317 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:36.140292    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:36.312821 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:36.312792    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:36.561345 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:36.561313    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:36.940168 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:36.940139    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8b8cfdc8-jc8vx_5a411cba-bd53-4f31-bfb9-098ec849238d/authorino/0.log"
Jul 28 20:33:42.775595 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:42.775559    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6s5zb_a9cb6c87-c689-4ed6-9ce2-4371cd243859/manager/0.log"
Jul 28 20:33:43.176579 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:43.176554    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856746f77b-q8wg9_fabb0aa1-2cac-421b-a8b4-8a86b39ea62f/copy-manifests/0.log"
Jul 28 20:33:43.183946 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:43.183917    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856746f77b-q8wg9_fabb0aa1-2cac-421b-a8b4-8a86b39ea62f/manager/0.log"
Jul 28 20:33:45.302731 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:45.302696    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8b8cfdc8-jc8vx_5a411cba-bd53-4f31-bfb9-098ec849238d/authorino/0.log"
Jul 28 20:33:45.997908 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:45.997869    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lwx8z_2c52c858-4373-4f1a-8cc3-4fc3f4037aac/limitador/0.log"
Jul 28 20:33:46.738388 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:46.738356    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77597fcbdb-sr9kb_fa904a59-c4b1-46aa-a3f6-f4df75ebc83c/kube-auth-proxy/0.log"
Jul 28 20:33:47.932585 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:47.932550    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vkjqk_15586622-87c4-4a53-97c8-18940ff00c50/main/0.log"
Jul 28 20:33:48.288649 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:48.288566    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-mbh7p_99347c96-f13b-4823-9b11-c32b3453fe47/main/0.log"
Jul 28 20:33:54.002577 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:54.002548    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:54.083375 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:54.083343    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:33:56.969395 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:56.969364    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9nkx9_464229a3-6aff-4f24-ba21-64b65a2182a4/global-pull-secret-syncer/0.log"
Jul 28 20:33:57.069877 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:57.069846    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log"
Jul 28 20:33:57.239542 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:33:57.239472    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-173.ec2.internal_415ae5ae7d5d3fc074b4ce415469a19e/haproxy/0.log"
Jul 28 20:34:01.232821 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:01.232789    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8b8cfdc8-jc8vx_5a411cba-bd53-4f31-bfb9-098ec849238d/authorino/0.log"
Jul 28 20:34:01.726093 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:01.726059    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lwx8z_2c52c858-4373-4f1a-8cc3-4fc3f4037aac/limitador/0.log"
Jul 28 20:34:03.938695 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:03.938666    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2fhj7_b72cac4a-87d7-4021-a509-ec0dbccacedc/node-exporter/0.log"
Jul 28 20:34:03.983845 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:03.983814    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2fhj7_b72cac4a-87d7-4021-a509-ec0dbccacedc/kube-rbac-proxy/0.log"
Jul 28 20:34:04.008729 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:04.008707    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2fhj7_b72cac4a-87d7-4021-a509-ec0dbccacedc/init-textfile/0.log"
Jul 28 20:34:06.181115 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.181081    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-qnll5_ea2f8539-0a89-4894-8bd6-73822815662e/networking-console-plugin/0.log"
Jul 28 20:34:06.893925 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.893688    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"]
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894406    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fafef7-0d56-4930-8d39-8e573a6a2a84" containerName="payload-pre-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894444    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fafef7-0d56-4930-8d39-8e573a6a2a84" containerName="payload-pre-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894492    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8eafb2d8-b01e-4a4b-b407-a6659afe8f31" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894505    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8eafb2d8-b01e-4a4b-b407-a6659afe8f31" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894526    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" containerName="main"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894536    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" containerName="main"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894547    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b2bed5-1263-4182-ac0f-c4243973b3f5" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894558    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b2bed5-1263-4182-ac0f-c4243973b3f5" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894579    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed43640-622a-4ab8-a73e-d75b2a85578e" containerName="payload-pre-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894589    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed43640-622a-4ab8-a73e-d75b2a85578e" containerName="payload-pre-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894603    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d65138-bff3-409f-bb7e-d91353fe6d6e" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894620    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d65138-bff3-409f-bb7e-d91353fe6d6e" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894630    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8894fd8-7ade-4ab0-ba50-31685d7f6d8e" containerName="payload-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894639    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8894fd8-7ade-4ab0-ba50-31685d7f6d8e" containerName="payload-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894665    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4fccce-07eb-4608-9704-bc4fd3071325" containerName="payload-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894673    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4fccce-07eb-4608-9704-bc4fd3071325" containerName="payload-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894682    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894690    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894853    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d65138-bff3-409f-bb7e-d91353fe6d6e" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894866    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ed43640-622a-4ab8-a73e-d75b2a85578e" containerName="payload-pre-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894876    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8eafb2d8-b01e-4a4b-b407-a6659afe8f31" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894920    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4fccce-07eb-4608-9704-bc4fd3071325" containerName="payload-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894932    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8894fd8-7ade-4ab0-ba50-31685d7f6d8e" containerName="payload-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894947    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2bc1188-8ae2-423b-8c88-fb7fc1759eb9" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894957    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b2bed5-1263-4182-ac0f-c4243973b3f5" containerName="maas-api"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894973    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fafef7-0d56-4930-8d39-8e573a6a2a84" containerName="payload-pre-processing"
Jul 28 20:34:06.897022 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.894986    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a39e285d-9f42-4f12-a7aa-4bf2bd7388dc" containerName="main"
Jul 28 20:34:06.899326 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.899300    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:06.901870 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.901846    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnmjx\"/\"kube-root-ca.crt\""
Jul 28 20:34:06.902156 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.902138    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bnmjx\"/\"openshift-service-ca.crt\""
Jul 28 20:34:06.902919 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.902879    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bnmjx\"/\"default-dockercfg-8nlqz\""
Jul 28 20:34:06.907247 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.907224    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"]
Jul 28 20:34:06.992408 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.992379    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-lib-modules\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:06.992566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.992461    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxtt\" (UniqueName: \"kubernetes.io/projected/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-kube-api-access-mqxtt\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:06.992566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.992491    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-proc\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:06.992566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.992516    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-sys\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:06.992566 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:06.992538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-podres\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.093674 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093646    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-proc\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.093798 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093694    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-sys\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.093798 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093725    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-podres\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.093798 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093759    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-proc\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.093798 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093760    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-lib-modules\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.094061 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-sys\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.094061 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093910    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-lib-modules\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.094061 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093961    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-podres\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.094061 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.093972    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxtt\" (UniqueName: \"kubernetes.io/projected/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-kube-api-access-mqxtt\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.103012 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.102982    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxtt\" (UniqueName: \"kubernetes.io/projected/8f38d6f2-1fa1-407d-99df-ffe09cf842cb-kube-api-access-mqxtt\") pod \"perf-node-gather-daemonset-c4lj7\" (UID: \"8f38d6f2-1fa1-407d-99df-ffe09cf842cb\") " pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.210705 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.210643    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.340847 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.340758    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"]
Jul 28 20:34:07.343417 ip-10-0-138-173 kubenswrapper[2555]: W0728 20:34:07.343396    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8f38d6f2_1fa1_407d_99df_ffe09cf842cb.slice/crio-f547aa280a47d8eae5a6f2c3970bd872970a97ea8c0eefd1458bde3e19a8c0a3 WatchSource:0}: Error finding container f547aa280a47d8eae5a6f2c3970bd872970a97ea8c0eefd1458bde3e19a8c0a3: Status 404 returned error can't find the container with id f547aa280a47d8eae5a6f2c3970bd872970a97ea8c0eefd1458bde3e19a8c0a3
Jul 28 20:34:07.964656 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.964619    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7" event={"ID":"8f38d6f2-1fa1-407d-99df-ffe09cf842cb","Type":"ContainerStarted","Data":"01f6558f3c0d1cf570dbfc45bb14a28fa03d8c7e0f00bf5c1b6f8fa29000b9a9"}
Jul 28 20:34:07.964837 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.964672    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7" event={"ID":"8f38d6f2-1fa1-407d-99df-ffe09cf842cb","Type":"ContainerStarted","Data":"f547aa280a47d8eae5a6f2c3970bd872970a97ea8c0eefd1458bde3e19a8c0a3"}
Jul 28 20:34:07.964837 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.964723    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:07.982309 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:07.982253    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7" podStartSLOduration=1.982234795 podStartE2EDuration="1.982234795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:34:07.981621096 +0000 UTC m=+3466.442379694" watchObservedRunningTime="2026-07-28 20:34:07.982234795 +0000 UTC m=+3466.442993395"
Jul 28 20:34:09.436857 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:09.436825    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tlzp2_ac2f6cd6-d537-45e7-b8a5-03034fd92b25/dns/0.log"
Jul 28 20:34:09.462274 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:09.462237    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tlzp2_ac2f6cd6-d537-45e7-b8a5-03034fd92b25/kube-rbac-proxy/0.log"
Jul 28 20:34:09.486993 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:09.486971    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-67xlq_666e0bbd-3122-4414-b121-88468187b07c/dns-node-resolver/0.log"
Jul 28 20:34:10.180292 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:10.180265    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pljjh_674936af-b442-43be-b18d-2677522519c3/node-ca/0.log"
Jul 28 20:34:11.312736 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:11.312700    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77597fcbdb-sr9kb_fa904a59-c4b1-46aa-a3f6-f4df75ebc83c/kube-auth-proxy/0.log"
Jul 28 20:34:12.446243 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:12.446212    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x7zrk_5d160f29-45be-4be6-8ff0-a0837ef292f7/serve-healthcheck-canary/0.log"
Jul 28 20:34:12.566407 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:12.566378    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:12.667575 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:12.667548    2555 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2569w_f9a6e2f7-b600-49dc-8f85-6622afec929e/konnectivity-agent/0.log" limit=1024
Jul 28 20:34:13.059219 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:13.059192    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gq6td_2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5/kube-rbac-proxy/0.log"
Jul 28 20:34:13.086874 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:13.086841    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gq6td_2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5/exporter/0.log"
Jul 28 20:34:13.113931 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:13.113911    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gq6td_2fc74456-ac91-4a6b-8f18-c5bb3dd7ecc5/extractor/0.log"
Jul 28 20:34:13.984747 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:13.984714    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bnmjx/perf-node-gather-daemonset-c4lj7"
Jul 28 20:34:16.401147 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:16.401118    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6s5zb_a9cb6c87-c689-4ed6-9ce2-4371cd243859/manager/0.log"
Jul 28 20:34:16.685755 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:16.685734    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856746f77b-q8wg9_fabb0aa1-2cac-421b-a8b4-8a86b39ea62f/manager/0.log"
Jul 28 20:34:16.712782 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:16.712758    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-856746f77b-q8wg9_fabb0aa1-2cac-421b-a8b4-8a86b39ea62f/copy-manifests/0.log"
Jul 28 20:34:17.651998 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:17.651968    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77597fcbdb-sr9kb_fa904a59-c4b1-46aa-a3f6-f4df75ebc83c/kube-auth-proxy/0.log"
Jul 28 20:34:18.058129 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:18.058105    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-cbcf9f5f9-gwcb6_5c69932b-85ed-4c83-9290-5e3205ccd6fc/manager/0.log"
Jul 28 20:34:19.494274 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:19.494246    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77597fcbdb-sr9kb_fa904a59-c4b1-46aa-a3f6-f4df75ebc83c/kube-auth-proxy/0.log"
Jul 28 20:34:23.357409 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:23.357371    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-w5wkj_8ed193d5-940f-4ec4-8c19-53f4ecef8e77/migrator/0.log"
Jul 28 20:34:23.379036 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:23.379004    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-w5wkj_8ed193d5-940f-4ec4-8c19-53f4ecef8e77/graceful-termination/0.log"
Jul 28 20:34:24.763153 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:24.763114    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4s7gc_6a4eb5e7-30db-4811-86f2-6870988e8487/kube-multus/0.log"
Jul 28 20:34:24.973882 ip-10-0-138-173 kubenswrapper[2555]: I0728 20:34:24.973856    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mj8cx_eddbfddc-1f2d-4857-8ad5-698e5fcf4d77/kube-multus-additional-cni-plugins/0.log"