Sep 16 18:39:27.526303 ip-10-0-141-134 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:39:27.876261 ip-10-0-141-134 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:27.876261 ip-10-0-141-134 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:39:27.876261 ip-10-0-141-134 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:27.876261 ip-10-0-141-134 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:39:27.876261 ip-10-0-141-134 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:39:27.877127 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.876997    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:39:27.881346 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881330    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:27.881346 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881345    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881350    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881353    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881356    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881359    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881361    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881364    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881367    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881370    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881372    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881375    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881379    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881382    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881384    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881387    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881390    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881392    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881403    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881406    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881409    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:27.881406 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881412    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881414    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881417    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881420    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881423    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881426    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881428    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881431    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881433    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881436    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881439    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881443    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881447    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881450    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881454    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881457    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881459    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881463    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881465    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881468    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:27.881884 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881470    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881473    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881475    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881479    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881482    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881484    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881487    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881489    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881491    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881494    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881497    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881500    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881502    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881505    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881510    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881513    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881515    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881518    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881521    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881523    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:27.882395 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881526    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881529    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881531    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881536    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881540    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881542    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881545    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881548    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881551    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881553    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881557    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881559    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881562    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881565    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881567    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881570    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881572    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881574    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881577    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:27.882886 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881580    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881582    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881585    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881587    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881590    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881593    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881596    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881598    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881602    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881604    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881607    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881610    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881613    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881616    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881619    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881622    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881625    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881627    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.881629    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:27.883359 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882323    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882330    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882334    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882337    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882340    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882343    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882346    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882349    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882351    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882354    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882357    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882360    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882363    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882365    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882368    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882371    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882373    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882377    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882380    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882382    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:27.883819 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882386    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882389    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882392    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882394    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882397    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882400    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882403    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882405    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882408    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882411    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882413    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882416    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882419    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882421    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882425    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882430    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882433    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882436    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882439    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:27.884333 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882442    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882445    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882448    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882450    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882453    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882456    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882459    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882462    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882464    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882467    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882469    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882472    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882476    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882479    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882481    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882486    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882490    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882493    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882496    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:27.884804 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882498    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882501    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882504    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882506    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882509    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882511    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882514    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882516    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882519    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882521    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882524    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882527    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882529    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882533    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882535    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882537    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882540    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882543    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882545    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882548    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:27.885284 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882551    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882555    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882558    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882562    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882565    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882567    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882570    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882572    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882575    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882578    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882580    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882583    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882585    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882588    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882591    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882593    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882596    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882598    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882601    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882604    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:27.885772 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882606    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.882998    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883004    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883007    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883011    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883014    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883017    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883020    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883022    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883025    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883028    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883031    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883034    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883038    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883042    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883045    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883048    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883051    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883053    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:27.886289 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883056    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883059    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883061    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883064    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883067    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883070    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883072    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883075    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883077    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883080    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883084    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883087    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883090    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883093    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883096    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883098    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883101    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883104    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883106    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:27.886753 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883109    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883111    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883114    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883116    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883124    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883127    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883130    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883133    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883136    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883140    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883142    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883145    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883148    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883150    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883153    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883155    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883158    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883160    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883163    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883165    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:27.887241 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883168    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883170    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883173    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883175    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883178    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883181    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883183    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883186    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883188    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883191    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883193    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883196    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883198    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883201    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883204    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883206    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883210    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883213    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883215    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883218    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:27.887728 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883220    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883224    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883227    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883229    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883232    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883234    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883237    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883240    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883243    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883245    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883248    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883251    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883253    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883256    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883258    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883261    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883263    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883266    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883268    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883271    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:27.888215 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883273    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.883276    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883349    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883380    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883387    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883391    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883396    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883399    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883405    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883409    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883413    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883416    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883420    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883424    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883428    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883431    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883434    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883437    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883440    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883443    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883447    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883450    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883453    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883456    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:39:27.888724 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883460    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883464    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883467    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883470    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883473    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883476    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883479    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883482    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883485    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883488    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883492    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883495    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883498    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883501    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883504    2559 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883508    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883514    2559 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883517    2559 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883520    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883523    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883526    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883530    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883533    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883537    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883540    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:39:27.889306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883547    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883550    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883553    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883556    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883559    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883562    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883565    2559 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883569    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883572    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883575    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883579    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883582    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883585    2559 flags.go:64] FLAG: --help="false"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883588    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-141-134.ec2.internal"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883591    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883594    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883597    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883600    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883603    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883606    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883609    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883612    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883615    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:39:27.889941 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883619    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883622    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883625    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883628    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883631    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883636    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883639    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883642    2559 flags.go:64] FLAG: --lock-file=""
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883645    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883648    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883651    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883656    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883659    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883661    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883664    2559 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883667    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883670    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883673    2559 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883676    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883680    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883683    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883687    2559 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883691    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883694    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883697    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:39:27.890578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883700    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883703    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883706    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883709    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883723    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883727    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883730    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883734    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883743    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883749    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883752    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883755    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883758    2559 flags.go:64] FLAG: --port="10250"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883762    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883765    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05b891e588293246e"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883768    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883771    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883774    2559 flags.go:64] FLAG: --register-node="true"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883777    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883780    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883783    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883786    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883789    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883793    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883796    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:39:27.891194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883798    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883801    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883804    2559 flags.go:64] FLAG: --runonce="false"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883807    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883810    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883813    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883816    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883819    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883822    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883826    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883829    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883832    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883835    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883837    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883843    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883846    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883855    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883858    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883864    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883867    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883871    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883877    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883880    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883883    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883886    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:39:27.891808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883889    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883892    2559 flags.go:64] FLAG: --v="2"
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883895    2559 flags.go:64] FLAG: --version="false"
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883899    2559 flags.go:64] FLAG: --vmodule=""
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883907    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.883910    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884057    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884061    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884064    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884067    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884070    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884073    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884076    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884079    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884082    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884085    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884088    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884091    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884094    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884097    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884100    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:27.892432 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884103    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884106    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884109    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884113    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884116    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884118    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884122    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884125    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884128    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884131    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884134    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884136    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884139    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884142    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884145    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884149    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884153    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884156    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884159    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884162    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:27.893032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884164    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884167    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884170    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884172    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884175    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884178    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884180    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884183    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884186    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884188    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884191    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884194    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884196    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884199    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884201    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884204    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884207    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884211    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884217    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884220    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:27.893558 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884223    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884225    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884229    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884232    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884235    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884238    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884242    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884245    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884248    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884251    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884254    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884257    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884259    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884262    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884265    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884267    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884270    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884272    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884275    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:27.894155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884278    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884280    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884283    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884286    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884288    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884291    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884294    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884296    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884299    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884302    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884304    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884308    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884311    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884314    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884316    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884319    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884322    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884324    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884327    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884330    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:27.894611 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884332    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884336    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884339    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884342    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.884345    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.884900    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.891747    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.891762    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.892370    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.892377    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892434    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892440    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892443    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892447    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892450    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:27.895130 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892453    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892456    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892459    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892462    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892464    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892467    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892470    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892472    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892475    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892478    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892480    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892484    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892488    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892491    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892494    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892498    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892501    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892503    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892506    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:27.895549 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892509    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892512    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892515    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892518    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892521    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892526    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892528    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892531    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892534    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892537    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892539    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892542    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892544    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892547    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892550    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892552    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892555    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892557    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892560    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892562    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:27.896024 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892565    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892567    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892570    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892574    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892578    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892581    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892584    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892587    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892589    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892592    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892594    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892597    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892600    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892602    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892607    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892611    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892614    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892617    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892621    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:27.896524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892623    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892626    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892628    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892631    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892634    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892636    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892639    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892642    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892644    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892647    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892650    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892653    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892655    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892658    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892660    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892663    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892666    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892669    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892671    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892674    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:27.897048 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892677    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892679    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892682    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892684    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892687    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892690    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892692    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892695    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892697    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892700    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892702    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892709    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892712    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892715    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892717    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:27.897526 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892720    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.892725    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892834    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892839    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892842    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892845    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892848    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892851    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892854    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892857    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892859    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892862    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892864    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892867    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:39:27.897903 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892870    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892873    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892875    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892878    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892880    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892883    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892886    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892888    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892891    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892893    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892896    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892898    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892901    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892904    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892906    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892909    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892912    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892914    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892917    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892919    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:39:27.898342 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892922    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892924    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892927    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892930    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892932    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892934    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892937    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892939    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892942    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892945    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892947    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892950    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892953    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892956    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892958    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892961    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892964    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892967    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892983    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892986    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:39:27.898837 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892988    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892991    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892994    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892997    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.892999    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893002    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893004    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893007    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893009    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893013    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893015    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893018    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893021    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893023    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893025    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893028    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893030    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893033    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893036    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893038    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:39:27.899398 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893041    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893044    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893047    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893049    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893053    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893057    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893060    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893063    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893065    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893069    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893071    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893074    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893076    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893079    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893081    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893084    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893086    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893089    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893091    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:39:27.900099 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893094    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893097    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893101    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893104    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893106    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893110    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893112    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:27.893115    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.893120    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.893219    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.895139    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.896071    2559 server.go:1019] "Starting client certificate rotation"
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.896167    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:27.900762 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.896722    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:39:27.901105 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.898323    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:27.902584 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.902557    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:39:27.910192 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.910176    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:39:27.912843 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.912823    2559 log.go:25] "Validated CRI v1 image API"
Sep 16 18:39:27.915231 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.915216    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:39:27.921355 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.921339    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 933a783d-9fd1-423c-a31e-1253f3d78090:/dev/nvme0n1p4 a2248977-a771-41fd-a9b8-90e835feebb0:/dev/nvme0n1p3]
Sep 16 18:39:27.921405 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.921355    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:39:27.926210 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.926105    2559 manager.go:216] Machine: {Timestamp:2026-09-16 18:39:27.925020909 +0000 UTC m=+0.293076811 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095647 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a06b5afead53027f8f6d565f9b26b SystemUUID:ec2a06b5-afea-d530-27f8-f6d565f9b26b BootID:a34f1761-d436-411a-91cd-ab1eacb9f927 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:f8:b0:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:f8:b0:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:82:1a:af:22:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:39:27.926210 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.926208    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:39:27.926315 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.926285    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:39:27.927322 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.927288    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:39:27.927453 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.927326    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-134.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:39:27.928460 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.928448    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:39:27.928499 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.928463    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:39:27.928499 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.928470    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:27.928499 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.928485    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:39:27.929300 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.929291    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:39:27.933100 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933089    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:27.933222 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933213    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:39:27.933255 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933247    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:39:27.933285 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933265    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:39:27.933312 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933284    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:39:27.933312 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933297    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:39:27.933312 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933306    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:39:27.933632 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933621    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:27.933675 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.933640    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:39:27.935643 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.935625    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:39:27.937377 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.937363    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:39:27.937435 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.937392    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:39:27.938833 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938822    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:39:27.938870 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938838    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:39:27.938870 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938844    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:39:27.938870 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938849    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:39:27.938870 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938858    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:39:27.938870 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938864    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:39:27.939031 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938879    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:39:27.939031 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938886    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:39:27.939031 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938893    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:39:27.939031 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938899    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:39:27.939031 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938907    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:39:27.939031 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.938912    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:39:27.939456 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.939444    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:39:27.939491 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.939457    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:39:27.940660 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.940642    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:27.943202 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.943179    2559 server.go:1268] "Started kubelet"
Sep 16 18:39:27.943306 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.943255    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:39:27.947091 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.946785    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:39:27.947184 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.947135    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:39:27.947522 ip-10-0-141-134 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:39:27.948174 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.948144    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:39:27.948643 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.948619    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:39:27.949158 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.948628    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-134.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:39:27.949247 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.948936    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:39:27.950497 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.950479    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:39:27.952018 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.952002    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:39:27.953808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.953785    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:27.954037 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.953123    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-134.ec2.internal.18d5e12320fc4419  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-134.ec2.internal,UID:ip-10-0-141-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-134.ec2.internal,},FirstTimestamp:2026-09-16 18:39:27.943148569 +0000 UTC m=+0.311204464,LastTimestamp:2026-09-16 18:39:27.943148569 +0000 UTC m=+0.311204464,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-134.ec2.internal,}"
Sep 16 18:39:27.954325 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.954304    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:39:27.954428 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.954409    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:27.955123 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955108    2559 factory.go:55] Registering systemd factory
Sep 16 18:39:27.955123 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955126    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:39:27.955314 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955298    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:39:27.955314 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955315    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:39:27.955407 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955322    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:39:27.955407 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955335    2559 factory.go:153] Registering CRI-O factory
Sep 16 18:39:27.955407 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955350    2559 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:39:27.955407 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955386    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:39:27.955407 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955396    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:39:27.955407 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955403    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:39:27.955619 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955420    2559 factory.go:103] Registering Raw factory
Sep 16 18:39:27.955619 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955438    2559 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:39:27.955734 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.955724    2559 manager.go:349] Starting recovery of all containers
Sep 16 18:39:27.965003 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.964959    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:39:27.965417 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.965356    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-134.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:39:27.965554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.965540    2559 manager.go:354] Recovery completed
Sep 16 18:39:27.970667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.970654    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:27.972249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.972231    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:27.972324 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.972263    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:27.972324 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.972276    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:27.972814 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.972802    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:39:27.972814 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.972813    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:39:27.972926 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.972826    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:39:27.972926 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.972805    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:27.973551 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.973539    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:27.973598 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.973561    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:27.973598 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:27.973574    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:27.978804 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.978741    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-134.ec2.internal.18d5e12322b84610  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-134.ec2.internal,UID:ip-10-0-141-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-134.ec2.internal,},FirstTimestamp:2026-09-16 18:39:27.972247056 +0000 UTC m=+0.340302947,LastTimestamp:2026-09-16 18:39:27.972247056 +0000 UTC m=+0.340302947,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-134.ec2.internal,}"
Sep 16 18:39:27.992541 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:27.992148    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-134.ec2.internal.18d5e12322b89ef4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-134.ec2.internal,UID:ip-10-0-141-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-134.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-134.ec2.internal,},FirstTimestamp:2026-09-16 18:39:27.972269812 +0000 UTC m=+0.340325708,LastTimestamp:2026-09-16 18:39:27.972269812 +0000 UTC m=+0.340325708,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-134.ec2.internal,}"
Sep 16 18:39:28.005936 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.005897    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l4xcm"
Sep 16 18:39:28.006586 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.006521    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-134.ec2.internal.18d5e12322b8c88c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-134.ec2.internal,UID:ip-10-0-141-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-134.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-134.ec2.internal,},FirstTimestamp:2026-09-16 18:39:27.97228046 +0000 UTC m=+0.340336353,LastTimestamp:2026-09-16 18:39:27.97228046 +0000 UTC m=+0.340336353,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-134.ec2.internal,}"
Sep 16 18:39:28.020922 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.020905    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l4xcm"
Sep 16 18:39:28.021460 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.021390    2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-134.ec2.internal.18d5e12322b84610\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-134.ec2.internal.18d5e12322b84610  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-134.ec2.internal,UID:ip-10-0-141-134.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-134.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-134.ec2.internal,},FirstTimestamp:2026-09-16 18:39:27.972247056 +0000 UTC m=+0.340302947,LastTimestamp:2026-09-16 18:39:27.973550947 +0000 UTC m=+0.341606838,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-134.ec2.internal,}"
Sep 16 18:39:28.054665 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.054646    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.120876 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.120847    2559 policy_none.go:49] "None policy: Start"
Sep 16 18:39:28.120876 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.120881    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:39:28.121031 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.120898    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:39:28.122028 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.122011    2559 policy_none.go:47] "Start"
Sep 16 18:39:28.147182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.146618    2559 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:39:28.147182 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.146654    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:39:28.147182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.146665    2559 server.go:87] "Starting device plugin registration server"
Sep 16 18:39:28.147182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.147119    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:39:28.147182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.147134    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:39:28.147451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.147238    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:39:28.147451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.147321    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:39:28.147451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.147330    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:39:28.147911 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.147886    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:39:28.148023 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.147945    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.172022 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.172001    2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-134.ec2.internal\" not found" node="ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.214682 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.214644    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:39:28.215899 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.215884    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:39:28.215964 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.215941    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:39:28.216034 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.215986    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:39:28.216079 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.216038    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:39:28.218551 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.218533    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:28.247548 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.247520    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:28.248564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.248543    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:28.248624 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.248588    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:28.248624 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.248601    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:28.248624 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.248625    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.260841 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.260823    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.260887 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.260849    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-134.ec2.internal\": node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.301565 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.301537    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.316725 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.316693    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"]
Sep 16 18:39:28.316816 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.316796    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:28.318318 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.318301    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:28.318400 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.318330    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:28.318400 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.318340    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:28.319720 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.319708    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:28.319847 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.319831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.319906 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.319867    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:28.320471 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.320451    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:28.320583 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.320475    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:28.320583 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.320493    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:28.320583 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.320509    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:28.320583 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.320529    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:28.320583 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.320544    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:28.321868 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.321855    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.321921 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.321880    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:39:28.322581 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.322566    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:39:28.322641 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.322593    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:39:28.322641 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.322603    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:39:28.355759 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.355733    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-134.ec2.internal\" not found" node="ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.356435 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.356407    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/987b3dcc62388b87e3077d94e8941aa6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-134.ec2.internal\" (UID: \"987b3dcc62388b87e3077d94e8941aa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.356510 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.356469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c55f20c7b3e5346d5e9e7e977f2e6b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal\" (UID: \"c55f20c7b3e5346d5e9e7e977f2e6b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.356510 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.356496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55f20c7b3e5346d5e9e7e977f2e6b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal\" (UID: \"c55f20c7b3e5346d5e9e7e977f2e6b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.360347 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.360323    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-134.ec2.internal\" not found" node="ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.402530 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.402462    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.456787 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.456753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/987b3dcc62388b87e3077d94e8941aa6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-134.ec2.internal\" (UID: \"987b3dcc62388b87e3077d94e8941aa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.456787 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.456782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c55f20c7b3e5346d5e9e7e977f2e6b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal\" (UID: \"c55f20c7b3e5346d5e9e7e977f2e6b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.457023 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.456802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55f20c7b3e5346d5e9e7e977f2e6b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal\" (UID: \"c55f20c7b3e5346d5e9e7e977f2e6b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.457023 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.456855    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/987b3dcc62388b87e3077d94e8941aa6-config\") pod \"kube-apiserver-proxy-ip-10-0-141-134.ec2.internal\" (UID: \"987b3dcc62388b87e3077d94e8941aa6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.457023 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.456862    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c55f20c7b3e5346d5e9e7e977f2e6b23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal\" (UID: \"c55f20c7b3e5346d5e9e7e977f2e6b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.457023 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.456861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c55f20c7b3e5346d5e9e7e977f2e6b23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal\" (UID: \"c55f20c7b3e5346d5e9e7e977f2e6b23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.503521 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.503491    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.604484 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.604430    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.658573 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.658500    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.663682 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.663661    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:28.704869 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.704823    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.808475 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.808433    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.896570 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.896537    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:39:28.897043 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.896688    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:39:28.908738 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:28.908668    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-134.ec2.internal\" not found"
Sep 16 18:39:28.957045 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:28.957018    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:29.025967 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.025918    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:34:28 +0000 UTC" deadline="2028-02-23 09:37:46.81856481 +0000 UTC"
Sep 16 18:39:29.025967 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.025963    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12590h58m17.792609147s"
Sep 16 18:39:29.055770 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.055742    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:29.067806 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.067788    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:29.069563 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.069544    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal"
Sep 16 18:39:29.110585 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.110560    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:39:29.233388 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.233360    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:29.319631 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:29.319595    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55f20c7b3e5346d5e9e7e977f2e6b23.slice/crio-541bb15adeaaba1e294404155d0603c5c4a626da470e36fa914cbb7ff4303d13 WatchSource:0}: Error finding container 541bb15adeaaba1e294404155d0603c5c4a626da470e36fa914cbb7ff4303d13: Status 404 returned error can't find the container with id 541bb15adeaaba1e294404155d0603c5c4a626da470e36fa914cbb7ff4303d13
Sep 16 18:39:29.320938 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:29.320917    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987b3dcc62388b87e3077d94e8941aa6.slice/crio-14645a482119358284517b54d59982a923e98f8c842de9e98e2dbabe8cfec2e2 WatchSource:0}: Error finding container 14645a482119358284517b54d59982a923e98f8c842de9e98e2dbabe8cfec2e2: Status 404 returned error can't find the container with id 14645a482119358284517b54d59982a923e98f8c842de9e98e2dbabe8cfec2e2
Sep 16 18:39:29.327130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.327113    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:39:29.522579 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.522491    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:29.934629 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.934537    2559 apiserver.go:52] "Watching apiserver"
Sep 16 18:39:29.942343 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.941259    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:39:29.942343 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.941880    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-mqdvl","kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal","openshift-cluster-node-tuning-operator/tuned-gdw6g","openshift-dns/node-resolver-qk4dv","openshift-image-registry/node-ca-8xtq5","openshift-multus/network-metrics-daemon-b44bb","kube-system/global-pull-secret-syncer-mlhxh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal","openshift-multus/multus-5qcp8","openshift-multus/multus-additional-cni-plugins-g2smj","openshift-network-diagnostics/network-check-target-7bl5l","openshift-network-operator/iptables-alerter-rfw5x","openshift-ovn-kubernetes/ovnkube-node-9kh6c"]
Sep 16 18:39:29.944667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.944641    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:29.948080 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.947139    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:29.948080 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.947948    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mv4z6\""
Sep 16 18:39:29.948421 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.948397    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:39:29.948603 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.948583    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:39:29.948814 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.948796    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:39:29.949714 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.949693    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:29.949819 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:29.949779    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:29.950675 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.950647    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgd4x\""
Sep 16 18:39:29.950886 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.950867    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:39:29.951449 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.951410    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:29.951678 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.951657    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:29.952902 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.952882    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:29.954931 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.954896    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:39:29.955386 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.955363    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:29.955884 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.955860    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:39:29.955969 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.955944    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6cmg\""
Sep 16 18:39:29.956030 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.955993    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:39:29.958065 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.958038    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:29.958160 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:29.958121    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:29.958328 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.958309    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:39:29.959382 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.959357    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:39:29.959600 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.959580    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:39:29.959882 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.959860    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mm4l2\""
Sep 16 18:39:29.961759 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.961736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:29.964300 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.964278    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:29.966517 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.966493    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:39:29.966800 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.966779    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bqw9q\""
Sep 16 18:39:29.967078 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.967061    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:39:29.967160 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.967101    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:29.967812 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.967794    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:39:29.968164 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968145    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v8lms\""
Sep 16 18:39:29.968456 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968438    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:39:29.968706 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-registration-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:29.968785 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-device-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:29.968785 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968746    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:29.968895 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9162f61b-6af4-40e3-b4c5-1c7704c48322-kubelet-config\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:29.968895 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6t8\" (UniqueName: \"kubernetes.io/projected/85cfc9e7-4f44-40c4-8080-daa1472cf544-kube-api-access-ll6t8\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:29.969000 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9836b02-e20b-469f-8db4-9835dcb15cfc-iptables-alerter-script\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:29.969000 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8w9f\" (UniqueName: \"kubernetes.io/projected/a9836b02-e20b-469f-8db4-9835dcb15cfc-kube-api-access-g8w9f\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:29.969000 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:29.969000 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.968991    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-hosts-file\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969013    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-tmp-dir\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac745a81-e369-4633-b72d-a2e3a6ca030d-host\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4k4\" (UniqueName: \"kubernetes.io/projected/ac745a81-e369-4633-b72d-a2e3a6ca030d-kube-api-access-cw4k4\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-sys-fs\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl7dz\" (UniqueName: \"kubernetes.io/projected/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-kube-api-access-gl7dz\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969178    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9836b02-e20b-469f-8db4-9835dcb15cfc-host-slash\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9162f61b-6af4-40e3-b4c5-1c7704c48322-dbus\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:29.969225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969226    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6m85\" (UniqueName: \"kubernetes.io/projected/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-kube-api-access-z6m85\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:29.969668 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969252    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac745a81-e369-4633-b72d-a2e3a6ca030d-serviceca\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:29.969668 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969311    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:29.969668 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:29.969668 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.969352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-socket-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:29.977017 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.976991    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:29.979129 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.979110    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:29.979223 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:29.979190    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:29.979292 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.979275    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:39:29.979535 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.979516    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:39:29.979608 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.979562    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:39:29.979875 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.979856    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zzzlq\""
Sep 16 18:39:29.979945 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.979890    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:39:29.980170 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.980146    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:39:29.980269 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.980202    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:39:29.980336 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.980147    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:39:29.980389 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.980367    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:39:29.981138 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.981117    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kfczg\""
Sep 16 18:39:29.982779 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.982758    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:39:29.982922 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.982904    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:39:29.983205 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.983188    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:39:29.983331 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.983311    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dqvhx\""
Sep 16 18:39:29.983441 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.983420    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:39:29.983640 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.983623    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:39:29.999068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:29.999037    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-49x7z"
Sep 16 18:39:30.014208 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.014187    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-49x7z"
Sep 16 18:39:30.056479 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.056437    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:39:30.069821 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.069788    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-env-overrides\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.069977 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.069847    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovnkube-script-lib\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.069977 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.069882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-sys-fs\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.069977 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.069908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl7dz\" (UniqueName: \"kubernetes.io/projected/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-kube-api-access-gl7dz\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.069977 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.069932    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-kubelet\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.069977 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.069963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-hostroot\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.070276 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070026    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-sys-fs\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.070276 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-socket-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.070276 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-device-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.070276 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.070276 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070241    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-system-cni-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-cni-bin\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-device-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070330    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef01009f-d147-475f-9115-275e01b03529-konnectivity-ca\") pod \"konnectivity-agent-mqdvl\" (UID: \"ef01009f-d147-475f-9115-275e01b03529\") " pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070358    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-cni-netd\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysctl-conf\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-lib-modules\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070420    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-socket-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-slash\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070461    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-cni-bin\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-etc-selinux\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.070524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-tmp-dir\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac745a81-e369-4633-b72d-a2e3a6ca030d-host\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4k4\" (UniqueName: \"kubernetes.io/projected/ac745a81-e369-4633-b72d-a2e3a6ca030d-kube-api-access-cw4k4\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-cni-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-k8s-cni-cncf-io\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-etc-kubernetes\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-tmp-dir\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070748    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-cnibin\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070778    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef01009f-d147-475f-9115-275e01b03529-agent-certs\") pod \"konnectivity-agent-mqdvl\" (UID: \"ef01009f-d147-475f-9115-275e01b03529\") " pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070819    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac745a81-e369-4633-b72d-a2e3a6ca030d-host\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysconfig\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5j4\" (UniqueName: \"kubernetes.io/projected/74a3ac79-2f44-4249-8c19-77427e9073a6-kube-api-access-7q5j4\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-kubelet\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-tmp\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.071088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.070970    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-cnibin\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071024    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-modprobe-d\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071092    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-systemd\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.071131    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-systemd\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071185    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-node-log\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071216    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-hosts-file\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-socket-dir-parent\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-netns\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071292    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-run\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-host\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-tuned\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.071434    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:30.571401354 +0000 UTC m=+2.939457243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-log-socket\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.072002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071564    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-hosts-file\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9162f61b-6af4-40e3-b4c5-1c7704c48322-dbus\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6m85\" (UniqueName: \"kubernetes.io/projected/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-kube-api-access-z6m85\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071629    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9162f61b-6af4-40e3-b4c5-1c7704c48322-dbus\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-conf-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-ovn\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-registration-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6t8\" (UniqueName: \"kubernetes.io/projected/85cfc9e7-4f44-40c4-8080-daa1472cf544-kube-api-access-ll6t8\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-cni-multus\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071842    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-var-lib-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9836b02-e20b-469f-8db4-9835dcb15cfc-iptables-alerter-script\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.071963    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-registration-dir\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072272    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxzf\" (UniqueName: \"kubernetes.io/projected/ffb6e126-98d0-45f8-869b-1e2563e38468-kube-api-access-bxxzf\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.072880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072301    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-systemd-units\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffb6e126-98d0-45f8-869b-1e2563e38468-cni-binary-copy\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072395    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-sys\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072421    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovnkube-config\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072471    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwkc\" (UniqueName: \"kubernetes.io/projected/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-kube-api-access-qcwkc\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072501    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9836b02-e20b-469f-8db4-9835dcb15cfc-host-slash\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9836b02-e20b-469f-8db4-9835dcb15cfc-iptables-alerter-script\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072546    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac745a81-e369-4633-b72d-a2e3a6ca030d-serviceca\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysctl-d\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.072647    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072689    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9162f61b-6af4-40e3-b4c5-1c7704c48322-kubelet-config\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072745    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-etc-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8w9f\" (UniqueName: \"kubernetes.io/projected/a9836b02-e20b-469f-8db4-9835dcb15cfc-kube-api-access-g8w9f\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072804    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-daemon-config\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-multus-certs\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.073700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-var-lib-kubelet\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072885    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dncpw\" (UniqueName: \"kubernetes.io/projected/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-kube-api-access-dncpw\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-system-cni-dir\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072966    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.072995    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovn-node-metrics-cert\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.073019    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-kubernetes\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.073061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-run-netns\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.073160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9836b02-e20b-469f-8db4-9835dcb15cfc-host-slash\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.073443    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret podName:9162f61b-6af4-40e3-b4c5-1c7704c48322 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:30.573416098 +0000 UTC m=+2.941471985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret") pod "global-pull-secret-syncer-mlhxh" (UID: "9162f61b-6af4-40e3-b4c5-1c7704c48322") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.073510    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9162f61b-6af4-40e3-b4c5-1c7704c48322-kubelet-config\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:30.074554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.073593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac745a81-e369-4633-b72d-a2e3a6ca030d-serviceca\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:30.113489 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.113193    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:39:30.119082 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.118892    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8w9f\" (UniqueName: \"kubernetes.io/projected/a9836b02-e20b-469f-8db4-9835dcb15cfc-kube-api-access-g8w9f\") pod \"iptables-alerter-rfw5x\" (UID: \"a9836b02-e20b-469f-8db4-9835dcb15cfc\") " pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:30.120320 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.120293    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4k4\" (UniqueName: \"kubernetes.io/projected/ac745a81-e369-4633-b72d-a2e3a6ca030d-kube-api-access-cw4k4\") pod \"node-ca-8xtq5\" (UID: \"ac745a81-e369-4633-b72d-a2e3a6ca030d\") " pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:30.120451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.120348    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl7dz\" (UniqueName: \"kubernetes.io/projected/02c4c009-2c2e-49b0-88c5-c3be7e9712cf-kube-api-access-gl7dz\") pod \"aws-ebs-csi-driver-node-fj2b5\" (UID: \"02c4c009-2c2e-49b0-88c5-c3be7e9712cf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.120451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.120359    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6t8\" (UniqueName: \"kubernetes.io/projected/85cfc9e7-4f44-40c4-8080-daa1472cf544-kube-api-access-ll6t8\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:30.122392 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.122366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6m85\" (UniqueName: \"kubernetes.io/projected/fe2346ee-9571-4d40-bc5d-cb7292a5efd3-kube-api-access-z6m85\") pod \"node-resolver-qk4dv\" (UID: \"fe2346ee-9571-4d40-bc5d-cb7292a5efd3\") " pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:30.173548 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.173548 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovn-node-metrics-cert\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-kubernetes\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-run-netns\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-env-overrides\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173646    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovnkube-script-lib\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173688    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-kubelet\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-hostroot\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-run-netns\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-system-cni-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-cni-bin\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173788    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef01009f-d147-475f-9115-275e01b03529-konnectivity-ca\") pod \"konnectivity-agent-mqdvl\" (UID: \"ef01009f-d147-475f-9115-275e01b03529\") " pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:30.173804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-cni-netd\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173834    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-kubernetes\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysctl-conf\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173866    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-lib-modules\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-slash\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-cni-bin\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-cni-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysctl-conf\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.173983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-k8s-cni-cncf-io\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-etc-kubernetes\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174114    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-cnibin\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174182    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef01009f-d147-475f-9115-275e01b03529-agent-certs\") pod \"konnectivity-agent-mqdvl\" (UID: \"ef01009f-d147-475f-9115-275e01b03529\") " pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysconfig\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5j4\" (UniqueName: \"kubernetes.io/projected/74a3ac79-2f44-4249-8c19-77427e9073a6-kube-api-access-7q5j4\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-kubelet\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.174410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174260    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-lib-modules\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-tmp\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-cnibin\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-modprobe-d\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174423    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-cni-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-systemd\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-systemd\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-node-log\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-socket-dir-parent\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-netns\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-run\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174608    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-host\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-tuned\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-k8s-cni-cncf-io\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174134    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-cni-bin\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.175271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-log-socket\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174690    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-log-socket\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-etc-kubernetes\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-conf-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174765    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-cnibin\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovnkube-script-lib\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174314    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-kubelet\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-slash\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174912    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-kubelet\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-hostroot\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-system-cni-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175100    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-cni-bin\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.174781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-ovn\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175147    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-cni-multus\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-cni-netd\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-modprobe-d\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-var-lib-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.176249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175251    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-systemd\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175299    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-var-lib-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-var-lib-cni-multus\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-host\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175447    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-run\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175505    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-ovn\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175552    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-systemd\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-node-log\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175657    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-socket-dir-parent\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175705    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-netns\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef01009f-d147-475f-9115-275e01b03529-konnectivity-ca\") pod \"konnectivity-agent-mqdvl\" (UID: \"ef01009f-d147-475f-9115-275e01b03529\") " pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175763    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysconfig\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxzf\" (UniqueName: \"kubernetes.io/projected/ffb6e126-98d0-45f8-869b-1e2563e38468-kube-api-access-bxxzf\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-env-overrides\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-systemd-units\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.177303 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175831    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74a3ac79-2f44-4249-8c19-77427e9073a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-conf-dir\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffb6e126-98d0-45f8-869b-1e2563e38468-cni-binary-copy\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-systemd-units\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175900    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-sys\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovnkube-config\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-sys\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwkc\" (UniqueName: \"kubernetes.io/projected/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-kube-api-access-qcwkc\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.175995    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysctl-d\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176020    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176046    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-etc-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-daemon-config\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176115    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-multus-certs\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176140    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-var-lib-kubelet\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176166    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dncpw\" (UniqueName: \"kubernetes.io/projected/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-kube-api-access-dncpw\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.178229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-system-cni-dir\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176270    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-host-run-multus-certs\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176325    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-var-lib-kubelet\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176527    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ffb6e126-98d0-45f8-869b-1e2563e38468-cni-binary-copy\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176567    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-run-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-etc-openvswitch\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176627    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74a3ac79-2f44-4249-8c19-77427e9073a6-system-cni-dir\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176648    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-sysctl-d\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176662    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ffb6e126-98d0-45f8-869b-1e2563e38468-cnibin\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.176682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ffb6e126-98d0-45f8-869b-1e2563e38468-multus-daemon-config\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.177626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovn-node-metrics-cert\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.179106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.179051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-ovnkube-config\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.179661 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.179322    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-etc-tuned\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.179901 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.179877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef01009f-d147-475f-9115-275e01b03529-agent-certs\") pod \"konnectivity-agent-mqdvl\" (UID: \"ef01009f-d147-475f-9115-275e01b03529\") " pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:30.180344 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.180320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-tmp\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.191486 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.191433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dncpw\" (UniqueName: \"kubernetes.io/projected/59d3db1c-1e32-4f3f-89f6-0049416f8f4c-kube-api-access-dncpw\") pod \"tuned-gdw6g\" (UID: \"59d3db1c-1e32-4f3f-89f6-0049416f8f4c\") " pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.194617 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.194330    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:30.194617 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.194356    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:30.194617 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.194369    2559 projected.go:196] Error preparing data for projected volume kube-api-access-gdc9q for pod openshift-network-diagnostics/network-check-target-7bl5l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:30.194617 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.194442    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q podName:e24d209a-c74c-4d64-9bef-2c7e32e1d7c6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:30.694423395 +0000 UTC m=+3.062479292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gdc9q" (UniqueName: "kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q") pod "network-check-target-7bl5l" (UID: "e24d209a-c74c-4d64-9bef-2c7e32e1d7c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:30.195384 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.195339    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwkc\" (UniqueName: \"kubernetes.io/projected/bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe-kube-api-access-qcwkc\") pod \"ovnkube-node-9kh6c\" (UID: \"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.196441 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.196412    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxzf\" (UniqueName: \"kubernetes.io/projected/ffb6e126-98d0-45f8-869b-1e2563e38468-kube-api-access-bxxzf\") pod \"multus-5qcp8\" (UID: \"ffb6e126-98d0-45f8-869b-1e2563e38468\") " pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.196667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.196650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5j4\" (UniqueName: \"kubernetes.io/projected/74a3ac79-2f44-4249-8c19-77427e9073a6-kube-api-access-7q5j4\") pod \"multus-additional-cni-plugins-g2smj\" (UID: \"74a3ac79-2f44-4249-8c19-77427e9073a6\") " pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.222527 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.222464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal" event={"ID":"987b3dcc62388b87e3077d94e8941aa6","Type":"ContainerStarted","Data":"14645a482119358284517b54d59982a923e98f8c842de9e98e2dbabe8cfec2e2"}
Sep 16 18:39:30.224225 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.224188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal" event={"ID":"c55f20c7b3e5346d5e9e7e977f2e6b23","Type":"ContainerStarted","Data":"541bb15adeaaba1e294404155d0603c5c4a626da470e36fa914cbb7ff4303d13"}
Sep 16 18:39:30.261107 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.261075    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5"
Sep 16 18:39:30.274934 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:30.274901    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c4c009_2c2e_49b0_88c5_c3be7e9712cf.slice/crio-38f4e8923b84efdfda5ce2910635c3bfb1c7fca37a503391d11684a36a0accf7 WatchSource:0}: Error finding container 38f4e8923b84efdfda5ce2910635c3bfb1c7fca37a503391d11684a36a0accf7: Status 404 returned error can't find the container with id 38f4e8923b84efdfda5ce2910635c3bfb1c7fca37a503391d11684a36a0accf7
Sep 16 18:39:30.279309 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.279280    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rfw5x"
Sep 16 18:39:30.290438 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:30.290411    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9836b02_e20b_469f_8db4_9835dcb15cfc.slice/crio-6c281afcd617701693bd4f24ae638cd9e4b454d7fdbaf9210e3a9b9d3ec28d8f WatchSource:0}: Error finding container 6c281afcd617701693bd4f24ae638cd9e4b454d7fdbaf9210e3a9b9d3ec28d8f: Status 404 returned error can't find the container with id 6c281afcd617701693bd4f24ae638cd9e4b454d7fdbaf9210e3a9b9d3ec28d8f
Sep 16 18:39:30.296822 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.296798    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qk4dv"
Sep 16 18:39:30.301948 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.301919    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8xtq5"
Sep 16 18:39:30.308571 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:30.308535    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2346ee_9571_4d40_bc5d_cb7292a5efd3.slice/crio-4fd44ae95b1efdc4b1c67844e38136265af68082c7756b66f7b9cd184b557ba4 WatchSource:0}: Error finding container 4fd44ae95b1efdc4b1c67844e38136265af68082c7756b66f7b9cd184b557ba4: Status 404 returned error can't find the container with id 4fd44ae95b1efdc4b1c67844e38136265af68082c7756b66f7b9cd184b557ba4
Sep 16 18:39:30.309384 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.309364    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:30.313425 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:30.313394    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac745a81_e369_4633_b72d_a2e3a6ca030d.slice/crio-b0558c63ea2e2704a60656310e60ef2002ad25b4adc106242d0f72b90061de3c WatchSource:0}: Error finding container b0558c63ea2e2704a60656310e60ef2002ad25b4adc106242d0f72b90061de3c: Status 404 returned error can't find the container with id b0558c63ea2e2704a60656310e60ef2002ad25b4adc106242d0f72b90061de3c
Sep 16 18:39:30.318883 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.318861    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gdw6g"
Sep 16 18:39:30.320443 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:30.320415    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef01009f_d147_475f_9115_275e01b03529.slice/crio-690695a343d5d40271bb466060579a4515dfbd0cfd188741d153b1cdf4ada1d6 WatchSource:0}: Error finding container 690695a343d5d40271bb466060579a4515dfbd0cfd188741d153b1cdf4ada1d6: Status 404 returned error can't find the container with id 690695a343d5d40271bb466060579a4515dfbd0cfd188741d153b1cdf4ada1d6
Sep 16 18:39:30.326266 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.326101    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5qcp8"
Sep 16 18:39:30.330179 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:30.330154    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d3db1c_1e32_4f3f_89f6_0049416f8f4c.slice/crio-cde8d137d645245faff547e7abfb324fa410a0b9120cb4c92f9bd03a1a2e3007 WatchSource:0}: Error finding container cde8d137d645245faff547e7abfb324fa410a0b9120cb4c92f9bd03a1a2e3007: Status 404 returned error can't find the container with id cde8d137d645245faff547e7abfb324fa410a0b9120cb4c92f9bd03a1a2e3007
Sep 16 18:39:30.335071 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.334941    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2smj"
Sep 16 18:39:30.337445 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:30.337416    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb6e126_98d0_45f8_869b_1e2563e38468.slice/crio-fc756d8839fff8dd7f88ca22870931a457c7107bbd1b3c7a56c5570649230eab WatchSource:0}: Error finding container fc756d8839fff8dd7f88ca22870931a457c7107bbd1b3c7a56c5570649230eab: Status 404 returned error can't find the container with id fc756d8839fff8dd7f88ca22870931a457c7107bbd1b3c7a56c5570649230eab
Sep 16 18:39:30.341980 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.341576    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:30.350460 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:39:30.350431    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a3ac79_2f44_4249_8c19_77427e9073a6.slice/crio-1fece05219a03f9fb5bf9aaf1eb1f06d85e3b42fa80d1a9bab997c1aacce2417 WatchSource:0}: Error finding container 1fece05219a03f9fb5bf9aaf1eb1f06d85e3b42fa80d1a9bab997c1aacce2417: Status 404 returned error can't find the container with id 1fece05219a03f9fb5bf9aaf1eb1f06d85e3b42fa80d1a9bab997c1aacce2417
Sep 16 18:39:30.442277 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.442205    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:39:30.579350 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.579312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:30.579527 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.579386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:30.579527 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.579481    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:30.579527 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.579505    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:30.579657 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.579554    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret podName:9162f61b-6af4-40e3-b4c5-1c7704c48322 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:31.579532988 +0000 UTC m=+3.947588880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret") pod "global-pull-secret-syncer-mlhxh" (UID: "9162f61b-6af4-40e3-b4c5-1c7704c48322") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:30.579657 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.579572    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:31.579564254 +0000 UTC m=+3.947620134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:30.780710 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:30.780636    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:30.780859 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.780800    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:30.780859 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.780825    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:30.780859 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.780839    2559 projected.go:196] Error preparing data for projected volume kube-api-access-gdc9q for pod openshift-network-diagnostics/network-check-target-7bl5l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:30.780979 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:30.780907    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q podName:e24d209a-c74c-4d64-9bef-2c7e32e1d7c6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:31.780886601 +0000 UTC m=+4.148942485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdc9q" (UniqueName: "kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q") pod "network-check-target-7bl5l" (UID: "e24d209a-c74c-4d64-9bef-2c7e32e1d7c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:31.017633 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.017583    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:30 +0000 UTC" deadline="2028-06-15 06:22:16.494854434 +0000 UTC"
Sep 16 18:39:31.017633 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.017632    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15299h42m45.477226684s"
Sep 16 18:39:31.216387 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.216298    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:31.216542 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.216450    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:31.216542 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.216520    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:31.216677 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.216652    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:31.228486 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.228376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5" event={"ID":"02c4c009-2c2e-49b0-88c5-c3be7e9712cf","Type":"ContainerStarted","Data":"38f4e8923b84efdfda5ce2910635c3bfb1c7fca37a503391d11684a36a0accf7"}
Sep 16 18:39:31.231515 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.231455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"57bfcdf9142856f2413618f7b837e14f4215df7243c7a949e2eb837eaf5aac46"}
Sep 16 18:39:31.233928 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.233877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8xtq5" event={"ID":"ac745a81-e369-4633-b72d-a2e3a6ca030d","Type":"ContainerStarted","Data":"b0558c63ea2e2704a60656310e60ef2002ad25b4adc106242d0f72b90061de3c"}
Sep 16 18:39:31.235612 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.235587    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qk4dv" event={"ID":"fe2346ee-9571-4d40-bc5d-cb7292a5efd3","Type":"ContainerStarted","Data":"4fd44ae95b1efdc4b1c67844e38136265af68082c7756b66f7b9cd184b557ba4"}
Sep 16 18:39:31.237643 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.237615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2smj" event={"ID":"74a3ac79-2f44-4249-8c19-77427e9073a6","Type":"ContainerStarted","Data":"1fece05219a03f9fb5bf9aaf1eb1f06d85e3b42fa80d1a9bab997c1aacce2417"}
Sep 16 18:39:31.239144 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.239118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5qcp8" event={"ID":"ffb6e126-98d0-45f8-869b-1e2563e38468","Type":"ContainerStarted","Data":"fc756d8839fff8dd7f88ca22870931a457c7107bbd1b3c7a56c5570649230eab"}
Sep 16 18:39:31.240626 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.240599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gdw6g" event={"ID":"59d3db1c-1e32-4f3f-89f6-0049416f8f4c","Type":"ContainerStarted","Data":"cde8d137d645245faff547e7abfb324fa410a0b9120cb4c92f9bd03a1a2e3007"}
Sep 16 18:39:31.244902 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.244870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mqdvl" event={"ID":"ef01009f-d147-475f-9115-275e01b03529","Type":"ContainerStarted","Data":"690695a343d5d40271bb466060579a4515dfbd0cfd188741d153b1cdf4ada1d6"}
Sep 16 18:39:31.247175 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.247147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rfw5x" event={"ID":"a9836b02-e20b-469f-8db4-9835dcb15cfc","Type":"ContainerStarted","Data":"6c281afcd617701693bd4f24ae638cd9e4b454d7fdbaf9210e3a9b9d3ec28d8f"}
Sep 16 18:39:31.588008 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.587835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:31.588008 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.587957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:31.588278 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.588110    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:31.588278 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.588178    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret podName:9162f61b-6af4-40e3-b4c5-1c7704c48322 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:33.588158302 +0000 UTC m=+5.956214199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret") pod "global-pull-secret-syncer-mlhxh" (UID: "9162f61b-6af4-40e3-b4c5-1c7704c48322") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:31.588688 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.588668    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:31.588781 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.588727    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:33.588710371 +0000 UTC m=+5.956766267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:31.790282 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:31.790244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:31.790497 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.790478    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:31.790561 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.790506    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:31.790561 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.790519    2559 projected.go:196] Error preparing data for projected volume kube-api-access-gdc9q for pod openshift-network-diagnostics/network-check-target-7bl5l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:31.790680 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:31.790576    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q podName:e24d209a-c74c-4d64-9bef-2c7e32e1d7c6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:33.79055743 +0000 UTC m=+6.158613316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdc9q" (UniqueName: "kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q") pod "network-check-target-7bl5l" (UID: "e24d209a-c74c-4d64-9bef-2c7e32e1d7c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:32.018823 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:32.018784    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:34:30 +0000 UTC" deadline="2028-05-05 10:33:40.684745079 +0000 UTC"
Sep 16 18:39:32.019271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:32.018825    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14319h54m8.665925304s"
Sep 16 18:39:32.219206 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:32.218562    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:32.219206 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:32.218711    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:33.216836 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:33.216727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:33.217785 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.216880    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:33.217785 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:33.217309    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:33.217785 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.217407    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:33.277759 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:33.277719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal" event={"ID":"c55f20c7b3e5346d5e9e7e977f2e6b23","Type":"ContainerStarted","Data":"bb3de455ca97792a6214a87178cd3764f09466a764538aaf0a401466c8c815be"}
Sep 16 18:39:33.281616 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:33.281580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal" event={"ID":"987b3dcc62388b87e3077d94e8941aa6","Type":"ContainerStarted","Data":"02445d2ea107b1283b21c0834261c9fcab8efd66cae30f456dc55c92426fb719"}
Sep 16 18:39:33.329854 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:33.329793    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-134.ec2.internal" podStartSLOduration=4.329769114 podStartE2EDuration="4.329769114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:33.329163672 +0000 UTC m=+5.697219578" watchObservedRunningTime="2026-09-16 18:39:33.329769114 +0000 UTC m=+5.697825022"
Sep 16 18:39:33.613120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:33.613012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:33.613120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:33.613101    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:33.613366 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.613265    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:33.613366 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.613327    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:37.613307193 +0000 UTC m=+9.981363080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:33.613787 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.613730    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:33.613787 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.613791    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret podName:9162f61b-6af4-40e3-b4c5-1c7704c48322 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:37.613774721 +0000 UTC m=+9.981830608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret") pod "global-pull-secret-syncer-mlhxh" (UID: "9162f61b-6af4-40e3-b4c5-1c7704c48322") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:33.815677 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:33.815638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:33.815864 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.815853    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:33.815923 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.815875    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:33.815923 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.815891    2559 projected.go:196] Error preparing data for projected volume kube-api-access-gdc9q for pod openshift-network-diagnostics/network-check-target-7bl5l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:33.816081 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:33.815955    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q podName:e24d209a-c74c-4d64-9bef-2c7e32e1d7c6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:37.815933126 +0000 UTC m=+10.183989008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdc9q" (UniqueName: "kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q") pod "network-check-target-7bl5l" (UID: "e24d209a-c74c-4d64-9bef-2c7e32e1d7c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:34.220769 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:34.220159    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:34.220769 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:34.220309    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:35.216915 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:35.216872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:35.217133 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:35.217022    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:35.217439 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:35.217417    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:35.217575 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:35.217532    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:35.290767 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:35.290726    2559 generic.go:356] "Generic (PLEG): container finished" podID="c55f20c7b3e5346d5e9e7e977f2e6b23" containerID="bb3de455ca97792a6214a87178cd3764f09466a764538aaf0a401466c8c815be" exitCode=0
Sep 16 18:39:35.291281 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:35.290791    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal" event={"ID":"c55f20c7b3e5346d5e9e7e977f2e6b23","Type":"ContainerDied","Data":"bb3de455ca97792a6214a87178cd3764f09466a764538aaf0a401466c8c815be"}
Sep 16 18:39:36.218756 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:36.217405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:36.218756 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:36.217626    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:37.217182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:37.217149    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:37.217615 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.217286    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:37.217709 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:37.217619    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:37.217762 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.217712    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:37.655873 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:37.655781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:37.655873 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:37.655864    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:37.656100 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.655931    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:37.656100 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.656009    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret podName:9162f61b-6af4-40e3-b4c5-1c7704c48322 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:45.655988389 +0000 UTC m=+18.024044287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret") pod "global-pull-secret-syncer-mlhxh" (UID: "9162f61b-6af4-40e3-b4c5-1c7704c48322") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:37.656100 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.656057    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:37.656277 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.656140    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:45.656126395 +0000 UTC m=+18.024182290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:37.858161 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:37.858125    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:37.858345 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.858329    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:37.858410 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.858354    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:37.858410 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.858367    2559 projected.go:196] Error preparing data for projected volume kube-api-access-gdc9q for pod openshift-network-diagnostics/network-check-target-7bl5l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:37.858525 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:37.858424    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q podName:e24d209a-c74c-4d64-9bef-2c7e32e1d7c6 nodeName:}" failed. No retries permitted until 2026-09-16 18:39:45.858406069 +0000 UTC m=+18.226461951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdc9q" (UniqueName: "kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q") pod "network-check-target-7bl5l" (UID: "e24d209a-c74c-4d64-9bef-2c7e32e1d7c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:38.218324 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:38.217834    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:38.218324 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:38.217955    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:39.216520 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:39.216424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:39.216676 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:39.216551    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:39.216676 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:39.216633    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:39.216787 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:39.216757    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:40.216595 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:40.216555    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:40.217062 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:40.216689    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:41.216599 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:41.216564    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:41.216599 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:41.216602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:41.217096 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:41.216664    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:41.217096 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:41.216803    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:42.216468 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:42.216431    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:42.216655 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:42.216572    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:43.217350 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:43.217125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:43.217758 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:43.217182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:43.217758 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:43.217463    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:43.217758 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:43.217549    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:44.217088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:44.217051    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:44.217270 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:44.217194    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:45.216581 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:45.216544    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:45.217039 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:45.216550    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:45.217039 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.216646    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:45.217039 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.216763    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:45.712694 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:45.712655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:45.712880 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:45.712731    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:45.712880 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.712832    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:45.712880 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.712863    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:45.713073 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.712913    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret podName:9162f61b-6af4-40e3-b4c5-1c7704c48322 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:01.71289344 +0000 UTC m=+34.080949324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret") pod "global-pull-secret-syncer-mlhxh" (UID: "9162f61b-6af4-40e3-b4c5-1c7704c48322") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:39:45.713073 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.712935    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:01.712924271 +0000 UTC m=+34.080980157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:39:45.915204 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:45.915168    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:45.915398 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.915315    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:39:45.915398 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.915335    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:39:45.915398 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.915343    2559 projected.go:196] Error preparing data for projected volume kube-api-access-gdc9q for pod openshift-network-diagnostics/network-check-target-7bl5l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:45.915398 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:45.915395    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q podName:e24d209a-c74c-4d64-9bef-2c7e32e1d7c6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:01.915379706 +0000 UTC m=+34.283435587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdc9q" (UniqueName: "kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q") pod "network-check-target-7bl5l" (UID: "e24d209a-c74c-4d64-9bef-2c7e32e1d7c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:39:46.216722 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:46.216692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:46.217177 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:46.216813    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:47.217081 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:47.217044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:47.217487 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:47.217045    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:47.217487 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:47.217166    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:47.217487 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:47.217251    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:48.217032 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:48.216996    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:48.217207 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:48.217118    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:49.216394 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.216180    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:49.216479 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.216187    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:49.216533 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:49.216470    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:49.216590 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:49.216531    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:49.318415 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.318337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5" event={"ID":"02c4c009-2c2e-49b0-88c5-c3be7e9712cf","Type":"ContainerStarted","Data":"2d3ab3347ba20955e74fea095e29c75897b3398992a58fa736ea06946a2b3c6d"}
Sep 16 18:39:49.321631 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.321607    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:39:49.322048 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.322021    2559 generic.go:356] "Generic (PLEG): container finished" podID="bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe" containerID="58ebbfc89da08e2558c12f9bb998b9e2677ab81544190c4c03ed08b88b40eab4" exitCode=1
Sep 16 18:39:49.322147 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.322093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"91d920cbba5540cf61935c959cc79f16faa59d31b0b399a201e0a1ff13da09d3"}
Sep 16 18:39:49.322202 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.322174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"6f4f77bf479ba455882700c23d4ead889552798e9544f9715fca78288acb3d11"}
Sep 16 18:39:49.322258 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.322229    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"f487c371713a1adaf9f7d097699855ccaf93e131ab87ea2f8ac13e3ec699d935"}
Sep 16 18:39:49.322322 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.322271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"01db91ea65258e4829e8f59355119966a47c268e76f5fc4b62d571f4e39e0d40"}
Sep 16 18:39:49.322322 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.322316    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerDied","Data":"58ebbfc89da08e2558c12f9bb998b9e2677ab81544190c4c03ed08b88b40eab4"}
Sep 16 18:39:49.322412 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.322364    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"da4931afc39bf4e6adc8e311c0a8dd37f6662b88e8eec7a7155e6420afdb4e65"}
Sep 16 18:39:49.323674 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.323652    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8xtq5" event={"ID":"ac745a81-e369-4633-b72d-a2e3a6ca030d","Type":"ContainerStarted","Data":"72e5334d4c3e7b0bf3c6d9ad07acfec371ec896f35636ea1c821fc9a80c17055"}
Sep 16 18:39:49.325276 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.325230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qk4dv" event={"ID":"fe2346ee-9571-4d40-bc5d-cb7292a5efd3","Type":"ContainerStarted","Data":"38d77d9be886ac8785c085a69a2b224d786b5d489441f6e448d323ab2978c796"}
Sep 16 18:39:49.327190 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.327156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal" event={"ID":"c55f20c7b3e5346d5e9e7e977f2e6b23","Type":"ContainerStarted","Data":"2bfd6094412f3849435d3f48602592da13bfda4c85efdcc9ea893bb3151a4608"}
Sep 16 18:39:49.329005 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.328985    2559 generic.go:356] "Generic (PLEG): container finished" podID="74a3ac79-2f44-4249-8c19-77427e9073a6" containerID="ac747dbf4db2a779f760f48701cb0f86638032f32e3b52af8dc8d1c5d40ed085" exitCode=0
Sep 16 18:39:49.329090 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.329058    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2smj" event={"ID":"74a3ac79-2f44-4249-8c19-77427e9073a6","Type":"ContainerDied","Data":"ac747dbf4db2a779f760f48701cb0f86638032f32e3b52af8dc8d1c5d40ed085"}
Sep 16 18:39:49.330849 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.330797    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5qcp8" event={"ID":"ffb6e126-98d0-45f8-869b-1e2563e38468","Type":"ContainerStarted","Data":"2186b38bf825da28dee0971964a52225a5ea1cb4737856ab833f49091927e1ae"}
Sep 16 18:39:49.332611 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.332589    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gdw6g" event={"ID":"59d3db1c-1e32-4f3f-89f6-0049416f8f4c","Type":"ContainerStarted","Data":"70fbb9a478464775d8bf8fda1a11673210b0de07c65dfe885bcbadd38b6a2b00"}
Sep 16 18:39:49.334002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.333981    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mqdvl" event={"ID":"ef01009f-d147-475f-9115-275e01b03529","Type":"ContainerStarted","Data":"b143badd040424f63005d0158000ceca38bb505857f9622710a0906fb4ce1d1c"}
Sep 16 18:39:49.358491 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.358440    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8xtq5" podStartSLOduration=2.072226041 podStartE2EDuration="20.35842184s" totalImagesPullingTime="18.286195799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:49.339806933 +0000 UTC m=+21.707862859" watchObservedRunningTime="2026-09-16 18:39:49.35842184 +0000 UTC m=+21.726477743"
Sep 16 18:39:49.358763 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.358726    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-134.ec2.internal" podStartSLOduration=20.358714238 podStartE2EDuration="20.358714238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:49.358113586 +0000 UTC m=+21.726169508" watchObservedRunningTime="2026-09-16 18:39:49.358714238 +0000 UTC m=+21.726770140"
Sep 16 18:39:49.402602 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.402557    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mqdvl" podStartSLOduration=2.123807297 podStartE2EDuration="20.402540366s" totalImagesPullingTime="18.278733069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:49.402191314 +0000 UTC m=+21.770247235" watchObservedRunningTime="2026-09-16 18:39:49.402540366 +0000 UTC m=+21.770596266"
Sep 16 18:39:49.402783 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.402753    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qk4dv" podStartSLOduration=2.111432289 podStartE2EDuration="20.402738493s" totalImagesPullingTime="18.291306204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:49.380078546 +0000 UTC m=+21.748134438" watchObservedRunningTime="2026-09-16 18:39:49.402738493 +0000 UTC m=+21.770794591"
Sep 16 18:39:49.428092 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.428039    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gdw6g" podStartSLOduration=2.157802103 podStartE2EDuration="20.428026471s" totalImagesPullingTime="18.270224368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:49.427885894 +0000 UTC m=+21.795941807" watchObservedRunningTime="2026-09-16 18:39:49.428026471 +0000 UTC m=+21.796082372"
Sep 16 18:39:49.486641 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:49.486395    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5qcp8" podStartSLOduration=1.953846248 podStartE2EDuration="20.486376627s" totalImagesPullingTime="18.532530379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:49.451156651 +0000 UTC m=+21.819212571" watchObservedRunningTime="2026-09-16 18:39:49.486376627 +0000 UTC m=+21.854432529"
Sep 16 18:39:50.044585 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:50.044554    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:39:50.156988 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:50.156802    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:39:50.044580165Z","UUID":"123b4e0a-07a1-47d3-9249-f1d2c5f7223a","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:39:50.158664 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:50.158640    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:39:50.158794 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:50.158678    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:39:50.216350 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:50.216307    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:50.216489 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:50.216452    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:50.338340 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:50.338300    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rfw5x" event={"ID":"a9836b02-e20b-469f-8db4-9835dcb15cfc","Type":"ContainerStarted","Data":"35b07d676142cedc3d0309ee7e04d02bbf1139e7e5fcbb184c35dac9a56ead28"}
Sep 16 18:39:50.340757 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:50.340717    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5" event={"ID":"02c4c009-2c2e-49b0-88c5-c3be7e9712cf","Type":"ContainerStarted","Data":"2c7321205b6d9a5f81cca7e57140407416ee12a01f3fd68d85c1dece349dfa8f"}
Sep 16 18:39:51.216350 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:51.216324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:51.216562 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:51.216443    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:51.216562 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:51.216511    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:51.216685 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:51.216630    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:51.345730 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:51.345561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5" event={"ID":"02c4c009-2c2e-49b0-88c5-c3be7e9712cf","Type":"ContainerStarted","Data":"ee4d8dbafac759f30ba5c974c9012ff6ad8b883f430fe115c3c629cec3a8280b"}
Sep 16 18:39:51.372774 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:51.372716    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rfw5x" podStartSLOduration=9.104062648 podStartE2EDuration="23.372697101s" totalImagesPullingTime="14.268634453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:50.356825339 +0000 UTC m=+22.724881267" watchObservedRunningTime="2026-09-16 18:39:51.372697101 +0000 UTC m=+23.740753005"
Sep 16 18:39:52.217015 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:52.216962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:52.217166 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:52.217134    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:52.352033 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:52.352002    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:39:52.352633 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:52.352609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"6bc2c8f73aa01ce9450c07fda30866dfcada2b64ea52f2ab675863f2dcaab6a2"}
Sep 16 18:39:52.547264 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:52.547185    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:52.547792 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:52.547775    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:52.575524 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:52.575471    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fj2b5" podStartSLOduration=4.191749974 podStartE2EDuration="24.575456785s" totalImagesPullingTime="20.383706811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:51.373256005 +0000 UTC m=+23.741311900" watchObservedRunningTime="2026-09-16 18:39:52.575456785 +0000 UTC m=+24.943512734"
Sep 16 18:39:53.216285 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:53.216240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:53.216285 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:53.216272    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:53.216513 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:53.216374    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:53.216570 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:53.216503    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:53.354870 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:53.354833    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:53.355368 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:53.355352    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mqdvl"
Sep 16 18:39:54.216740 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:54.216566    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:54.216906 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:54.216824    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:54.360273 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:54.360246    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:39:54.360958 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:54.360659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"869050c0501bd9be1dabfc2b9f08870fc177891cfd8ac2eab4d91fa654db2c47"}
Sep 16 18:39:54.360958 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:54.360893    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:54.361167 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:54.361148    2559 scope.go:117] "RemoveContainer" containerID="58ebbfc89da08e2558c12f9bb998b9e2677ab81544190c4c03ed08b88b40eab4"
Sep 16 18:39:54.363122 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:54.363096    2559 generic.go:356] "Generic (PLEG): container finished" podID="74a3ac79-2f44-4249-8c19-77427e9073a6" containerID="ae874c614c3514e44d3d8b20a328dfe68e0c9e571c8915497a7c2214998f161e" exitCode=0
Sep 16 18:39:54.363209 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:54.363173    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2smj" event={"ID":"74a3ac79-2f44-4249-8c19-77427e9073a6","Type":"ContainerDied","Data":"ae874c614c3514e44d3d8b20a328dfe68e0c9e571c8915497a7c2214998f161e"}
Sep 16 18:39:54.377808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:54.377787    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:55.216499 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.216333    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:55.216640 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.216405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:55.216640 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:55.216568    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:55.216702 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:55.216671    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:55.368101 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.368009    2559 generic.go:356] "Generic (PLEG): container finished" podID="74a3ac79-2f44-4249-8c19-77427e9073a6" containerID="291e0ad4cf11354e5d792be13f44a228e3785cd56bc939fad112262d0b3ded1f" exitCode=0
Sep 16 18:39:55.368568 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.368094    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2smj" event={"ID":"74a3ac79-2f44-4249-8c19-77427e9073a6","Type":"ContainerDied","Data":"291e0ad4cf11354e5d792be13f44a228e3785cd56bc939fad112262d0b3ded1f"}
Sep 16 18:39:55.372104 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.372087    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:39:55.372506 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.372487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" event={"ID":"bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe","Type":"ContainerStarted","Data":"6b00b9e2bb11754995f150acaadb789bd0574b665e10ef1617e99c89c0fe09b3"}
Sep 16 18:39:55.372597 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.372580    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:55.372658 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.372647    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:55.387962 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.387945    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:39:55.421556 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.421510    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c" podStartSLOduration=8.152616885 podStartE2EDuration="26.421492796s" totalImagesPullingTime="18.268875911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:39:55.421493296 +0000 UTC m=+27.789549219" watchObservedRunningTime="2026-09-16 18:39:55.421492796 +0000 UTC m=+27.789548697"
Sep 16 18:39:55.556677 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.556648    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlhxh"]
Sep 16 18:39:55.556838 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.556777    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:55.556946 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:55.556925    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:55.561150 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.561126    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b44bb"]
Sep 16 18:39:55.561270 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.561234    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:55.561354 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:55.561337    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:55.561842 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.561823    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7bl5l"]
Sep 16 18:39:55.561900 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:55.561891    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:55.561990 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:55.561960    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:56.376856 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:56.376826    2559 generic.go:356] "Generic (PLEG): container finished" podID="74a3ac79-2f44-4249-8c19-77427e9073a6" containerID="a68235da15ef9d4dc429ea9810e5eb2111fad87df0f4abfe8ae5e65da8e533cc" exitCode=0
Sep 16 18:39:56.377231 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:56.376920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2smj" event={"ID":"74a3ac79-2f44-4249-8c19-77427e9073a6","Type":"ContainerDied","Data":"a68235da15ef9d4dc429ea9810e5eb2111fad87df0f4abfe8ae5e65da8e533cc"}
Sep 16 18:39:57.216668 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:57.216633    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:57.216844 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:57.216680    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:57.216844 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:57.216737    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:57.216844 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:57.216752    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:57.216844 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:57.216803    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:57.217073 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:57.216903    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:39:59.216459 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:59.216427    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:39:59.216965 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:59.216485    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:39:59.216965 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:39:59.216599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:39:59.216965 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:59.216616    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:39:59.216965 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:59.216674    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:39:59.216965 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:39:59.216784    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:40:01.217329 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.217155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:40:01.217739 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.217236    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:40:01.217739 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.217427    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:40:01.217739 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.217245    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:40:01.217739 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.217485    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7bl5l" podUID="e24d209a-c74c-4d64-9bef-2c7e32e1d7c6"
Sep 16 18:40:01.217739 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.217570    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mlhxh" podUID="9162f61b-6af4-40e3-b4c5-1c7704c48322"
Sep 16 18:40:01.377987 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.377933    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-134.ec2.internal" event="NodeReady"
Sep 16 18:40:01.378162 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.378084    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:40:01.392317 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.392284    2559 generic.go:356] "Generic (PLEG): container finished" podID="74a3ac79-2f44-4249-8c19-77427e9073a6" containerID="d9b5e2e208c1926864ecefab4012be67900adc53473754b3cc071109713870a0" exitCode=0
Sep 16 18:40:01.392538 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.392345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2smj" event={"ID":"74a3ac79-2f44-4249-8c19-77427e9073a6","Type":"ContainerDied","Data":"d9b5e2e208c1926864ecefab4012be67900adc53473754b3cc071109713870a0"}
Sep 16 18:40:01.416305 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.416268    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-675b454989-mb9cg"]
Sep 16 18:40:01.431039 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.430966    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"]
Sep 16 18:40:01.431241 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.431205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.434147 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.434121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:40:01.434866 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.434467    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tm7mf\""
Sep 16 18:40:01.434866 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.434755    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:40:01.435042 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.434965    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:40:01.443902 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.443875    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"]
Sep 16 18:40:01.444947 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.444750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"
Sep 16 18:40:01.449328 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.448026    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:40:01.449328 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.448207    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:40:01.450130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.450072    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"]
Sep 16 18:40:01.452120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.451534    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.452239 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.452193    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-h259p\""
Sep 16 18:40:01.452461 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.452324    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:40:01.452562 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.452464    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 18:40:01.452838 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.452820    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:40:01.456754 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.456730    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 18:40:01.466390 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.466370    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"]
Sep 16 18:40:01.466505 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.466407    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"]
Sep 16 18:40:01.466505 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.466426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675b454989-mb9cg"]
Sep 16 18:40:01.466505 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.466451    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4tzv7"]
Sep 16 18:40:01.466640 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.466504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.469314 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.469252    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:40:01.469411 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.469333    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:40:01.469756 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.469727    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:40:01.470761 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.470738    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:40:01.480758 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.480739    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xqbwk"]
Sep 16 18:40:01.480881 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.480864    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.483796 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.483773    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:40:01.483878 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.483795    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-24cbm\""
Sep 16 18:40:01.483878 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.483811    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:40:01.497157 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.497139    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"]
Sep 16 18:40:01.497233 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.497167    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4tzv7"]
Sep 16 18:40:01.497233 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.497190    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xqbwk"]
Sep 16 18:40:01.497300 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.497244    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xqbwk"
Sep 16 18:40:01.500130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.500109    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:40:01.500223 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.500191    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:40:01.500309 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.500296    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:40:01.500397 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.500384    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7zs8w\""
Sep 16 18:40:01.527107 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527085    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz\" (UID: \"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"
Sep 16 18:40:01.527269 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527110    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7f06edd-789b-43ab-91d3-e83bb86a928d-ca-trust-extracted\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.527269 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-trusted-ca\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.527269 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-installation-pull-secrets\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.527436 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527383    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.527491 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-certificates\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.527549 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4jcv\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-kube-api-access-x4jcv\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.527549 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527532    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-image-registry-private-configuration\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.527649 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvt9\" (UniqueName: \"kubernetes.io/projected/641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686-kube-api-access-pbvt9\") pod \"managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz\" (UID: \"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"
Sep 16 18:40:01.527649 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.527605    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-bound-sa-token\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.628503 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwhd\" (UniqueName: \"kubernetes.io/projected/d3342a0b-c9c9-46bc-beab-8fd7574ef11b-kube-api-access-5mwhd\") pod \"ingress-canary-xqbwk\" (UID: \"d3342a0b-c9c9-46bc-beab-8fd7574ef11b\") " pod="openshift-ingress-canary/ingress-canary-xqbwk"
Sep 16 18:40:01.628642 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628513    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.628642 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-tmp-dir\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.628642 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628583    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvt9\" (UniqueName: \"kubernetes.io/projected/641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686-kube-api-access-pbvt9\") pod \"managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz\" (UID: \"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"
Sep 16 18:40:01.628642 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dea3e2ac-1b6a-4571-9c66-8014c23d2710-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.628834 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92rq\" (UniqueName: \"kubernetes.io/projected/a2d4b9c7-33e3-4647-938c-541b1b2b9648-kube-api-access-v92rq\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.628834 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-bound-sa-token\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.628834 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8bf\" (UniqueName: \"kubernetes.io/projected/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-kube-api-access-9x8bf\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.628968 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz\" (UID: \"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"
Sep 16 18:40:01.628968 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7f06edd-789b-43ab-91d3-e83bb86a928d-ca-trust-extracted\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.628968 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-trusted-ca\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.628968 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-installation-pull-secrets\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.628968 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea3e2ac-1b6a-4571-9c66-8014c23d2710-tmp\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.628968 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3342a0b-c9c9-46bc-beab-8fd7574ef11b-cert\") pod \"ingress-canary-xqbwk\" (UID: \"d3342a0b-c9c9-46bc-beab-8fd7574ef11b\") " pod="openshift-ingress-canary/ingress-canary-xqbwk"
Sep 16 18:40:01.628968 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.628995    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629067    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-hub\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629123    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d4b9c7-33e3-4647-938c-541b1b2b9648-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.629141    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27zp\" (UniqueName: \"kubernetes.io/projected/dea3e2ac-1b6a-4571-9c66-8014c23d2710-kube-api-access-g27zp\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.629163    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b454989-mb9cg: secret "image-registry-tls" not found
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.629262    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls podName:d7f06edd-789b-43ab-91d3-e83bb86a928d nodeName:}" failed. No retries permitted until 2026-09-16 18:40:02.129242042 +0000 UTC m=+34.497297934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls") pod "image-registry-675b454989-mb9cg" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d") : secret "image-registry-tls" not found
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629188    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-ca\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.629323 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7f06edd-789b-43ab-91d3-e83bb86a928d-ca-trust-extracted\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.629706 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629327    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-config-volume\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.629706 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-certificates\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.629706 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4jcv\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-kube-api-access-x4jcv\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.629706 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-image-registry-private-configuration\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.629706 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.629924 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.629856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-certificates\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.630354 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.630334    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-trusted-ca\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.632968 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.632946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-installation-pull-secrets\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.633060 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.633020    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-image-registry-private-configuration\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.633147 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.633132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz\" (UID: \"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"
Sep 16 18:40:01.637444 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.637419    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-bound-sa-token\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.637607 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.637590    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvt9\" (UniqueName: \"kubernetes.io/projected/641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686-kube-api-access-pbvt9\") pod \"managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz\" (UID: \"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"
Sep 16 18:40:01.637674 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.637657    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4jcv\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-kube-api-access-x4jcv\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:01.730900 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.730825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-hub\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.730900 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.730857    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d4b9c7-33e3-4647-938c-541b1b2b9648-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.730900 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.730878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27zp\" (UniqueName: \"kubernetes.io/projected/dea3e2ac-1b6a-4571-9c66-8014c23d2710-kube-api-access-g27zp\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.730900 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.730894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-ca\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.731251 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.730923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:40:01.731251 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-config-volume\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.731251 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.731251 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731105    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwhd\" (UniqueName: \"kubernetes.io/projected/d3342a0b-c9c9-46bc-beab-8fd7574ef11b-kube-api-access-5mwhd\") pod \"ingress-canary-xqbwk\" (UID: \"d3342a0b-c9c9-46bc-beab-8fd7574ef11b\") " pod="openshift-ingress-canary/ingress-canary-xqbwk"
Sep 16 18:40:01.731251 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.731251 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-tmp-dir\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.731251 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dea3e2ac-1b6a-4571-9c66-8014c23d2710-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.731571 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.731037    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:01.731571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731348    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92rq\" (UniqueName: \"kubernetes.io/projected/a2d4b9c7-33e3-4647-938c-541b1b2b9648-kube-api-access-v92rq\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.731571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8bf\" (UniqueName: \"kubernetes.io/projected/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-kube-api-access-9x8bf\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.731571 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.731173    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:01.731571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:40:01.731571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea3e2ac-1b6a-4571-9c66-8014c23d2710-tmp\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.731571 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.731511    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls podName:3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:02.231480759 +0000 UTC m=+34.599536655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls") pod "dns-default-4tzv7" (UID: "3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:01.731571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731549    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3342a0b-c9c9-46bc-beab-8fd7574ef11b-cert\") pod \"ingress-canary-xqbwk\" (UID: \"d3342a0b-c9c9-46bc-beab-8fd7574ef11b\") " pod="openshift-ingress-canary/ingress-canary-xqbwk"
Sep 16 18:40:01.731958 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.731958 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731625    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d4b9c7-33e3-4647-938c-541b1b2b9648-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.731958 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731625    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-config-volume\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.731958 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.731693    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret podName:9162f61b-6af4-40e3-b4c5-1c7704c48322 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.731674953 +0000 UTC m=+66.099730846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret") pod "global-pull-secret-syncer-mlhxh" (UID: "9162f61b-6af4-40e3-b4c5-1c7704c48322") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:40:01.731958 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-tmp-dir\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.731958 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.731932    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dea3e2ac-1b6a-4571-9c66-8014c23d2710-tmp\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.732258 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.732084    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:01.732258 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.732129    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.732113973 +0000 UTC m=+66.100169867 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:40:01.734075 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.734047    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-ca\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.734269 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.734246    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3342a0b-c9c9-46bc-beab-8fd7574ef11b-cert\") pod \"ingress-canary-xqbwk\" (UID: \"d3342a0b-c9c9-46bc-beab-8fd7574ef11b\") " pod="openshift-ingress-canary/ingress-canary-xqbwk"
Sep 16 18:40:01.734443 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.734422    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.734515 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.734482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.734573 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.734501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a2d4b9c7-33e3-4647-938c-541b1b2b9648-hub\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.734625 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.734605    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/dea3e2ac-1b6a-4571-9c66-8014c23d2710-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.739261 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.739239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27zp\" (UniqueName: \"kubernetes.io/projected/dea3e2ac-1b6a-4571-9c66-8014c23d2710-kube-api-access-g27zp\") pod \"klusterlet-addon-workmgr-7d44b6bbcd-fczz5\" (UID: \"dea3e2ac-1b6a-4571-9c66-8014c23d2710\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.743526 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.743499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8bf\" (UniqueName: \"kubernetes.io/projected/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-kube-api-access-9x8bf\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:01.743820 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.743800    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwhd\" (UniqueName: \"kubernetes.io/projected/d3342a0b-c9c9-46bc-beab-8fd7574ef11b-kube-api-access-5mwhd\") pod \"ingress-canary-xqbwk\" (UID: \"d3342a0b-c9c9-46bc-beab-8fd7574ef11b\") " pod="openshift-ingress-canary/ingress-canary-xqbwk"
Sep 16 18:40:01.744021 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.743990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92rq\" (UniqueName: \"kubernetes.io/projected/a2d4b9c7-33e3-4647-938c-541b1b2b9648-kube-api-access-v92rq\") pod \"cluster-proxy-proxy-agent-7975f87cd4-qtpm6\" (UID: \"a2d4b9c7-33e3-4647-938c-541b1b2b9648\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.771587 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.771557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"
Sep 16 18:40:01.779729 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.779706    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:01.796325 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.796313    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:40:01.833465 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.833433    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xqbwk"
Sep 16 18:40:01.933711 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:01.933681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:40:01.933879 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.933857    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:40:01.933936 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.933883    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:40:01.933936 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.933895    2559 projected.go:196] Error preparing data for projected volume kube-api-access-gdc9q for pod openshift-network-diagnostics/network-check-target-7bl5l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:01.934026 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:01.933946    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q podName:e24d209a-c74c-4d64-9bef-2c7e32e1d7c6 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.933932041 +0000 UTC m=+66.301987921 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gdc9q" (UniqueName: "kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q") pod "network-check-target-7bl5l" (UID: "e24d209a-c74c-4d64-9bef-2c7e32e1d7c6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:40:02.083944 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.083868    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz"]
Sep 16 18:40:02.087191 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.087162    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"]
Sep 16 18:40:02.088109 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:40:02.088081    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod641e9dd1_e8eb_4f1d_ac46_e3f89d6dd686.slice/crio-f82efb41402310fbb3a97ec3ee0432310b752499a4b6e3e4c1f7d76c32f404cd WatchSource:0}: Error finding container f82efb41402310fbb3a97ec3ee0432310b752499a4b6e3e4c1f7d76c32f404cd: Status 404 returned error can't find the container with id f82efb41402310fbb3a97ec3ee0432310b752499a4b6e3e4c1f7d76c32f404cd
Sep 16 18:40:02.089649 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:40:02.089621    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea3e2ac_1b6a_4571_9c66_8014c23d2710.slice/crio-1b174a9d185169f011cc3ba2d312a4e9b90f1696d43f3439fa8679fd2cd77685 WatchSource:0}: Error finding container 1b174a9d185169f011cc3ba2d312a4e9b90f1696d43f3439fa8679fd2cd77685: Status 404 returned error can't find the container with id 1b174a9d185169f011cc3ba2d312a4e9b90f1696d43f3439fa8679fd2cd77685
Sep 16 18:40:02.135222 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.135191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:02.135348 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:02.135312    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:02.135348 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:02.135330    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b454989-mb9cg: secret "image-registry-tls" not found
Sep 16 18:40:02.135415 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:02.135386    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls podName:d7f06edd-789b-43ab-91d3-e83bb86a928d nodeName:}" failed. No retries permitted until 2026-09-16 18:40:03.135367882 +0000 UTC m=+35.503423777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls") pod "image-registry-675b454989-mb9cg" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d") : secret "image-registry-tls" not found
Sep 16 18:40:02.141451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.141428    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"]
Sep 16 18:40:02.145029 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:40:02.145005    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2d4b9c7_33e3_4647_938c_541b1b2b9648.slice/crio-1b0cd231e7a285ce9e706435ddb24f2f82b5d1cf5bcbd2f3344aaddb720a6449 WatchSource:0}: Error finding container 1b0cd231e7a285ce9e706435ddb24f2f82b5d1cf5bcbd2f3344aaddb720a6449: Status 404 returned error can't find the container with id 1b0cd231e7a285ce9e706435ddb24f2f82b5d1cf5bcbd2f3344aaddb720a6449
Sep 16 18:40:02.152307 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.152285    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xqbwk"]
Sep 16 18:40:02.154994 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:40:02.154955    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3342a0b_c9c9_46bc_beab_8fd7574ef11b.slice/crio-9312990eda4a08b5282b4d58bcb82893a2b0c1a5509951fb07fb39f2ab6f5a72 WatchSource:0}: Error finding container 9312990eda4a08b5282b4d58bcb82893a2b0c1a5509951fb07fb39f2ab6f5a72: Status 404 returned error can't find the container with id 9312990eda4a08b5282b4d58bcb82893a2b0c1a5509951fb07fb39f2ab6f5a72
Sep 16 18:40:02.236542 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.236512    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:02.237258 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:02.236660    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:02.237258 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:02.236715    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls podName:3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:03.236700456 +0000 UTC m=+35.604756340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls") pod "dns-default-4tzv7" (UID: "3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:02.396042 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.395781    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xqbwk" event={"ID":"d3342a0b-c9c9-46bc-beab-8fd7574ef11b","Type":"ContainerStarted","Data":"9312990eda4a08b5282b4d58bcb82893a2b0c1a5509951fb07fb39f2ab6f5a72"}
Sep 16 18:40:02.396956 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.396932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz" event={"ID":"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686","Type":"ContainerStarted","Data":"f82efb41402310fbb3a97ec3ee0432310b752499a4b6e3e4c1f7d76c32f404cd"}
Sep 16 18:40:02.398034 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.398014    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" event={"ID":"a2d4b9c7-33e3-4647-938c-541b1b2b9648","Type":"ContainerStarted","Data":"1b0cd231e7a285ce9e706435ddb24f2f82b5d1cf5bcbd2f3344aaddb720a6449"}
Sep 16 18:40:02.399087 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.399068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5" event={"ID":"dea3e2ac-1b6a-4571-9c66-8014c23d2710","Type":"ContainerStarted","Data":"1b174a9d185169f011cc3ba2d312a4e9b90f1696d43f3439fa8679fd2cd77685"}
Sep 16 18:40:02.402023 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.402004    2559 generic.go:356] "Generic (PLEG): container finished" podID="74a3ac79-2f44-4249-8c19-77427e9073a6" containerID="fc4d4208c2f0534bdf4f5f85cb9eff7e0c30756a917fcb5c88eb4fe72722a220" exitCode=0
Sep 16 18:40:02.402096 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:02.402032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2smj" event={"ID":"74a3ac79-2f44-4249-8c19-77427e9073a6","Type":"ContainerDied","Data":"fc4d4208c2f0534bdf4f5f85cb9eff7e0c30756a917fcb5c88eb4fe72722a220"}
Sep 16 18:40:03.144843 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.144803    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:03.145173 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:03.144997    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:03.145173 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:03.145013    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b454989-mb9cg: secret "image-registry-tls" not found
Sep 16 18:40:03.145173 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:03.145069    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls podName:d7f06edd-789b-43ab-91d3-e83bb86a928d nodeName:}" failed. No retries permitted until 2026-09-16 18:40:05.145051594 +0000 UTC m=+37.513107491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls") pod "image-registry-675b454989-mb9cg" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d") : secret "image-registry-tls" not found
Sep 16 18:40:03.216377 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.216341    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:40:03.216562 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.216387    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:40:03.216562 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.216341    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:40:03.220173 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.220023    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:03.220370 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.220320    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:03.221862 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.221840    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rmdqb\""
Sep 16 18:40:03.222099 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.222082    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l6kxj\""
Sep 16 18:40:03.222348 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.222328    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:03.222561 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.222545    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:03.246775 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.246223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:03.246775 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:03.246362    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:03.246775 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:03.246444    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls podName:3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:05.246422758 +0000 UTC m=+37.614478652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls") pod "dns-default-4tzv7" (UID: "3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:03.414249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.414127    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2smj" event={"ID":"74a3ac79-2f44-4249-8c19-77427e9073a6","Type":"ContainerStarted","Data":"c91dfdc6a94008993bcf176550282c3d83a5a43873ef74cef4e0fa46d6d9bdef"}
Sep 16 18:40:03.606949 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.606863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:03.725709 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:03.725673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:05.166345 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:05.166309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:05.166787 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:05.166464    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:05.166787 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:05.166484    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b454989-mb9cg: secret "image-registry-tls" not found
Sep 16 18:40:05.166787 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:05.166541    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls podName:d7f06edd-789b-43ab-91d3-e83bb86a928d nodeName:}" failed. No retries permitted until 2026-09-16 18:40:09.166528207 +0000 UTC m=+41.534584099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls") pod "image-registry-675b454989-mb9cg" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d") : secret "image-registry-tls" not found
Sep 16 18:40:05.268027 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:05.267422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:05.268027 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:05.267590    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:05.268027 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:05.267656    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls podName:3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:09.267635734 +0000 UTC m=+41.635691637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls") pod "dns-default-4tzv7" (UID: "3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:06.460086 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:06.460053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:06.639475 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:06.639433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:07.010630 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:07.010591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:07.113666 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:07.113639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:08.122517 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:08.122475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:08.216340 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:08.216295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:09.201515 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:09.201474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:09.202003 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:09.201627    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:09.202003 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:09.201647    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b454989-mb9cg: secret "image-registry-tls" not found
Sep 16 18:40:09.202003 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:09.201702    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls podName:d7f06edd-789b-43ab-91d3-e83bb86a928d nodeName:}" failed. No retries permitted until 2026-09-16 18:40:17.201684533 +0000 UTC m=+49.569740412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls") pod "image-registry-675b454989-mb9cg" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d") : secret "image-registry-tls" not found
Sep 16 18:40:09.302480 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:09.302443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:09.302653 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:09.302576    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:09.302653 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:09.302630    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls podName:3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:17.302615553 +0000 UTC m=+49.670671431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls") pod "dns-default-4tzv7" (UID: "3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:11.327585 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.327552    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:11.430402 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.430365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:11.446077 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.446041    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" event={"ID":"a2d4b9c7-33e3-4647-938c-541b1b2b9648","Type":"ContainerStarted","Data":"6756d1d58d9585e125bb6f213d9e52f7832b0d668d6947840c094aa87ed053ca"}
Sep 16 18:40:11.446242 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.446085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" event={"ID":"a2d4b9c7-33e3-4647-938c-541b1b2b9648","Type":"ContainerStarted","Data":"6fdbe443a1b77d23f58f937478bee3285348c80cf085f9b24f73123d71615dff"}
Sep 16 18:40:11.446242 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.446108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" event={"ID":"a2d4b9c7-33e3-4647-938c-541b1b2b9648","Type":"ContainerStarted","Data":"2c480333804fd21fc976b2c7a1dcd5d656b8071d37f0fe16e6471593813a7dee"}
Sep 16 18:40:11.447593 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.447571    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5" event={"ID":"dea3e2ac-1b6a-4571-9c66-8014c23d2710","Type":"ContainerStarted","Data":"8b667df73006464edeb4be343f75b076c426e5f73d2ece70105219de3641ce30"}
Sep 16 18:40:11.447702 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.447641    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:11.449289 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.449259    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xqbwk" event={"ID":"d3342a0b-c9c9-46bc-beab-8fd7574ef11b","Type":"ContainerStarted","Data":"86489dd7ed8e68a2508a7a1bc5c7daf0ee8664627f06f70ff18d185719452a97"}
Sep 16 18:40:11.449653 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.449636    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:40:11.450735 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.450713    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz" event={"ID":"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686","Type":"ContainerStarted","Data":"4a3037cbf21b44556990d666b03dc813ac7c4b9518045ac1a19c1431a8bee877"}
Sep 16 18:40:11.468342 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.468300    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g2smj" podStartSLOduration=14.143537441 podStartE2EDuration="42.468287195s" totalImagesPullingTime="28.324749754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:03.450689257 +0000 UTC m=+35.818745183" watchObservedRunningTime="2026-09-16 18:40:11.468287195 +0000 UTC m=+43.836343104"
Sep 16 18:40:11.487134 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.487094    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" podStartSLOduration=3.611392751 podStartE2EDuration="12.487079752s" totalImagesPullingTime="8.875687001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:11.468678688 +0000 UTC m=+43.836734599" watchObservedRunningTime="2026-09-16 18:40:11.487079752 +0000 UTC m=+43.855135653"
Sep 16 18:40:11.487359 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.487340    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz" podStartSLOduration=3.554717519 podStartE2EDuration="12.487335548s" totalImagesPullingTime="8.932618029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:11.486064736 +0000 UTC m=+43.854120638" watchObservedRunningTime="2026-09-16 18:40:11.487335548 +0000 UTC m=+43.855391448"
Sep 16 18:40:11.507884 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.507837    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5" podStartSLOduration=3.559425681 podStartE2EDuration="12.507826293s" totalImagesPullingTime="8.948400612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:11.507290817 +0000 UTC m=+43.875346735" watchObservedRunningTime="2026-09-16 18:40:11.507826293 +0000 UTC m=+43.875882229"
Sep 16 18:40:11.531040 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.530933    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xqbwk" podStartSLOduration=1.664632925 podStartE2EDuration="10.530916187s" totalImagesPullingTime="8.866283262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:11.530037177 +0000 UTC m=+43.898093099" watchObservedRunningTime="2026-09-16 18:40:11.530916187 +0000 UTC m=+43.898972123"
Sep 16 18:40:11.815879 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.815794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:11.888642 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:11.888605    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:13.166149 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:13.166110    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:13.266441 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:13.266397    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:15.908414 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:15.908367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:15.989301 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:15.989271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:17.254665 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:17.254623    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:17.255082 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:17.254779    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:17.255082 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:17.254797    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b454989-mb9cg: secret "image-registry-tls" not found
Sep 16 18:40:17.255082 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:17.254851    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls podName:d7f06edd-789b-43ab-91d3-e83bb86a928d nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.254834571 +0000 UTC m=+65.622890482 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls") pod "image-registry-675b454989-mb9cg" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d") : secret "image-registry-tls" not found
Sep 16 18:40:17.355759 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:17.355726    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:17.355915 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:17.355891    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:17.356027 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:17.356015    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls podName:3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93 nodeName:}" failed. No retries permitted until 2026-09-16 18:40:33.355994575 +0000 UTC m=+65.724050455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls") pod "dns-default-4tzv7" (UID: "3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:27.389426 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:27.389395    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9kh6c"
Sep 16 18:40:27.617155 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:27.617117    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:27.703954 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:27.703834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:32.035439 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:32.035353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:32.129993 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:32.129938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:33.154097 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.154066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:33.252670 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.252611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:33.275316 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.275288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:40:33.275450 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:33.275431    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:40:33.275489 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:33.275451    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b454989-mb9cg: secret "image-registry-tls" not found
Sep 16 18:40:33.275522 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:33.275512    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls podName:d7f06edd-789b-43ab-91d3-e83bb86a928d nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.275497711 +0000 UTC m=+97.643553594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls") pod "image-registry-675b454989-mb9cg" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d") : secret "image-registry-tls" not found
Sep 16 18:40:33.375736 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.375698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:40:33.375893 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:33.375847    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:40:33.375937 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:33.375912    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls podName:3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:05.375896942 +0000 UTC m=+97.743952841 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls") pod "dns-default-4tzv7" (UID: "3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93") : secret "dns-default-metrics-tls" not found
Sep 16 18:40:33.778813 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.778779    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:40:33.779006 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.778841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:40:33.781775 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.781753    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:40:33.781863 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.781757    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:40:33.789343 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:33.789325    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:40:33.789403 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:40:33.789385    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:37.789369666 +0000 UTC m=+130.157425558 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : secret "metrics-daemon-secret" not found
Sep 16 18:40:33.792299 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.792282    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9162f61b-6af4-40e3-b4c5-1c7704c48322-original-pull-secret\") pod \"global-pull-secret-syncer-mlhxh\" (UID: \"9162f61b-6af4-40e3-b4c5-1c7704c48322\") " pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:40:33.833190 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.833167    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mlhxh"
Sep 16 18:40:33.980046 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.980003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:40:33.983402 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.983380    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:40:33.993802 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:33.993762    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:40:34.003468 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:34.003441    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdc9q\" (UniqueName: \"kubernetes.io/projected/e24d209a-c74c-4d64-9bef-2c7e32e1d7c6-kube-api-access-gdc9q\") pod \"network-check-target-7bl5l\" (UID: \"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6\") " pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:40:34.107817 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:34.107677    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mlhxh"]
Sep 16 18:40:34.110217 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:40:34.110178    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9162f61b_6af4_40e3_b4c5_1c7704c48322.slice/crio-3feb70d2ec5d04c5c31339331bfdca1629cca8e15d5a84384567b08f4999e490 WatchSource:0}: Error finding container 3feb70d2ec5d04c5c31339331bfdca1629cca8e15d5a84384567b08f4999e490: Status 404 returned error can't find the container with id 3feb70d2ec5d04c5c31339331bfdca1629cca8e15d5a84384567b08f4999e490
Sep 16 18:40:34.146572 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:34.146542    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rmdqb\""
Sep 16 18:40:34.153673 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:34.153654    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:40:34.432659 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:34.432582    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7bl5l"]
Sep 16 18:40:34.436155 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:40:34.436123    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24d209a_c74c_4d64_9bef_2c7e32e1d7c6.slice/crio-7178b9529f6ee91bb707b3b7eaf52759b7e0fd34da62863494e27df3a4e30629 WatchSource:0}: Error finding container 7178b9529f6ee91bb707b3b7eaf52759b7e0fd34da62863494e27df3a4e30629: Status 404 returned error can't find the container with id 7178b9529f6ee91bb707b3b7eaf52759b7e0fd34da62863494e27df3a4e30629
Sep 16 18:40:34.519294 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:34.519260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlhxh" event={"ID":"9162f61b-6af4-40e3-b4c5-1c7704c48322","Type":"ContainerStarted","Data":"3feb70d2ec5d04c5c31339331bfdca1629cca8e15d5a84384567b08f4999e490"}
Sep 16 18:40:34.520408 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:34.520379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7bl5l" event={"ID":"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6","Type":"ContainerStarted","Data":"7178b9529f6ee91bb707b3b7eaf52759b7e0fd34da62863494e27df3a4e30629"}
Sep 16 18:40:39.540360 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:39.540322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mlhxh" event={"ID":"9162f61b-6af4-40e3-b4c5-1c7704c48322","Type":"ContainerStarted","Data":"edf76a505df29adccc5120b3579386237afef35420716c6e6a81e960933cef88"}
Sep 16 18:40:39.542089 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:39.542062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7bl5l" event={"ID":"e24d209a-c74c-4d64-9bef-2c7e32e1d7c6","Type":"ContainerStarted","Data":"2e6f66cf9e38e4460bec7d3892b779742139d4d3fa66ebbf7bd78161f1f1ab11"}
Sep 16 18:40:39.542230 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:39.542191    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:40:39.599368 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:39.599278    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mlhxh" podStartSLOduration=65.529403469 podStartE2EDuration="1m10.599260715s" totalImagesPullingTime="5.069857246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:39.570430958 +0000 UTC m=+71.938486878" watchObservedRunningTime="2026-09-16 18:40:39.599260715 +0000 UTC m=+71.967316633"
Sep 16 18:40:39.599698 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:39.599658    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7bl5l" podStartSLOduration=65.86493915 podStartE2EDuration="1m10.59964513s" totalImagesPullingTime="4.73470598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:39.599414987 +0000 UTC m=+71.967470866" watchObservedRunningTime="2026-09-16 18:40:39.59964513 +0000 UTC m=+71.967701034"
Sep 16 18:40:41.805924 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:41.805887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:40:41.899308 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:40:41.899262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:05.319262 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:05.319222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:41:05.319666 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:05.319357    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:41:05.319666 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:05.319369    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-675b454989-mb9cg: secret "image-registry-tls" not found
Sep 16 18:41:05.319666 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:05.319422    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls podName:d7f06edd-789b-43ab-91d3-e83bb86a928d nodeName:}" failed. No retries permitted until 2026-09-16 18:42:09.319408244 +0000 UTC m=+161.687464123 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls") pod "image-registry-675b454989-mb9cg" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d") : secret "image-registry-tls" not found
Sep 16 18:41:05.419812 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:05.419735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:41:05.419942 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:05.419879    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:41:05.419942 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:05.419940    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls podName:3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93 nodeName:}" failed. No retries permitted until 2026-09-16 18:42:09.419923617 +0000 UTC m=+161.787979496 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls") pod "dns-default-4tzv7" (UID: "3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93") : secret "dns-default-metrics-tls" not found
Sep 16 18:41:10.547936 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:10.547905    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7bl5l"
Sep 16 18:41:32.779192 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:32.779128    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:33.959807 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:33.959782    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.393023 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:34.392929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:34.495460 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:34.495410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:36.744119 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:36.744084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:36.893619 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:36.893579    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:37.050817 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:37.050730    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:37.135577 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:37.135532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:37.775696 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:37.775665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:37.845963 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:37.845930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:41:37.846145 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:37.846074    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:41:37.846183 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:37.846147    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs podName:85cfc9e7-4f44-40c4-8080-daa1472cf544 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:39.846130084 +0000 UTC m=+252.214185963 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs") pod "network-metrics-daemon-b44bb" (UID: "85cfc9e7-4f44-40c4-8080-daa1472cf544") : secret "metrics-daemon-secret" not found
Sep 16 18:41:37.867342 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:37.867307    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:38.762546 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.762507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:38.916235 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.916206    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-67ssr"]
Sep 16 18:41:38.918200 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.918183    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:38.940435 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.940411    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:41:38.940622 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.940605    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:41:38.941429 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.941408    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:41:38.941602 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.941589    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:41:38.941650 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.941611    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-blflj\""
Sep 16 18:41:38.952827 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.952803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:38.952887 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.952831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:38.952887 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.952847    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4rpt\" (UniqueName: \"kubernetes.io/projected/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-api-access-v4rpt\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:38.952887 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.952865    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a577c313-7fc7-4356-96bc-e9efeb8c0591-data-volume\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:38.952998 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.952950    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a577c313-7fc7-4356-96bc-e9efeb8c0591-crio-socket\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:38.972173 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:38.972151    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-67ssr"]
Sep 16 18:41:39.053754 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.053679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.053754 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.053718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.053754 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.053741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4rpt\" (UniqueName: \"kubernetes.io/projected/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-api-access-v4rpt\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.054011 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:39.053815    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:39.054011 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:39.053828    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:39.054011 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.053862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a577c313-7fc7-4356-96bc-e9efeb8c0591-data-volume\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.054011 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:39.053878    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:39.553862023 +0000 UTC m=+131.921917929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:39.054011 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:39.053943    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:39.553920322 +0000 UTC m=+131.921976213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:39.054230 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.054031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a577c313-7fc7-4356-96bc-e9efeb8c0591-crio-socket\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.054230 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.054160    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a577c313-7fc7-4356-96bc-e9efeb8c0591-crio-socket\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.054293 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.054255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a577c313-7fc7-4356-96bc-e9efeb8c0591-data-volume\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.083099 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.083078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4rpt\" (UniqueName: \"kubernetes.io/projected/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-api-access-v4rpt\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.310521 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.310442    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:39.485668 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.485634    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:39.558175 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.558142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.558256 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:39.558181    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:39.558297 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:39.558260    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:39.558343 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:39.558335    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:39.558385 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:39.558337    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:40.558321235 +0000 UTC m=+132.926377114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:39.558438 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:39.558396    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:40.558384278 +0000 UTC m=+132.926440157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:40.566744 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:40.566704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:40.566744 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:40.566744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:40.567227 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:40.566823    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:40.567227 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:40.566838    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:40.567227 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:40.566887    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.566871142 +0000 UTC m=+134.934927025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:40.567388 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:40.567295    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:42.567092537 +0000 UTC m=+134.935148435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:41.588683 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:41.588654    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qk4dv_fe2346ee-9571-4d40-bc5d-cb7292a5efd3/dns-node-resolver/0.log"
Sep 16 18:41:41.906541 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:41.906459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:42.388420 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:42.388395    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8xtq5_ac745a81-e369-4633-b72d-a2e3a6ca030d/node-ca/0.log"
Sep 16 18:41:42.581652 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:42.581595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:42.581652 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:42.581656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:42.581897 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:42.581719    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.581897 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:42.581785    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:46.581768504 +0000 UTC m=+138.949824383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:42.581897 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:42.581795    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:42.581897 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:42.581846    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:46.581831183 +0000 UTC m=+138.949887063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:43.073634 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:43.073599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.168274 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:43.168237    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.281538 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:43.281501    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.446637 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:43.446541    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xqbwk_d3342a0b-c9c9-46bc-beab-8fd7574ef11b/serve-healthcheck-canary/0.log"
Sep 16 18:41:43.446824 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:43.446769    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.608388 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:43.608352    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:43.767070 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:43.767038    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.533741 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:44.533698    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.604410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:44.604367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:44.935394 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:44.935311    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.349654 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:45.349615    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.550461 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:45.550425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:45.939770 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:45.939728    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.614138 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:46.614093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:46.614138 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:46.614142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:46.614586 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:46.614224    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:41:46.614586 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:46.614286    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:54.61426949 +0000 UTC m=+146.982325374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : configmap "kube-rbac-proxy" not found
Sep 16 18:41:46.614586 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:46.614324    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:46.614586 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:41:46.614381    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls podName:a577c313-7fc7-4356-96bc-e9efeb8c0591 nodeName:}" failed. No retries permitted until 2026-09-16 18:41:54.614368389 +0000 UTC m=+146.982424287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67ssr" (UID: "a577c313-7fc7-4356-96bc-e9efeb8c0591") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:41:46.771613 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:46.771576    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:46.878315 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:46.878223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.024059 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:47.024028    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.222834 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:47.222801    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.378632 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:47.378577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.590755 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:47.590673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.704145 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:47.704113    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:47.793780 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:47.793740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:48.574480 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:48.574450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.033695 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:49.033657    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.186019 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:49.185985    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.373028 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:49.372934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.489280 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:49.489244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.615789 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:49.615752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.766920 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:49.766886    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:49.883799 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:49.883765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:50.761723 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:50.761686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:51.173592 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:51.173510    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:51.376922 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:51.376887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:51.476734 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:51.476697    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:51.599759 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:51.599725    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:51.968166 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:51.968127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:52.541818 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:52.541781    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:53.162098 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:53.162060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:53.346246 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:53.346199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:53.547650 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:53.547595    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:53.747664 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:53.747628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:41:54.681982 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:54.681944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:54.682377 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:54.682008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:54.682494 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:54.682472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a577c313-7fc7-4356-96bc-e9efeb8c0591-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:54.684222 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:54.684202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a577c313-7fc7-4356-96bc-e9efeb8c0591-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67ssr\" (UID: \"a577c313-7fc7-4356-96bc-e9efeb8c0591\") " pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:54.826264 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:54.826230    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67ssr"
Sep 16 18:41:55.101391 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:55.101361    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-67ssr"]
Sep 16 18:41:55.104220 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:41:55.104183    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda577c313_7fc7_4356_96bc_e9efeb8c0591.slice/crio-d1889ed3c4a84ca6f20ede2e379ac028fde07fdcc4612138af9efda0106af823 WatchSource:0}: Error finding container d1889ed3c4a84ca6f20ede2e379ac028fde07fdcc4612138af9efda0106af823: Status 404 returned error can't find the container with id d1889ed3c4a84ca6f20ede2e379ac028fde07fdcc4612138af9efda0106af823
Sep 16 18:41:55.764400 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:55.764368    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67ssr" event={"ID":"a577c313-7fc7-4356-96bc-e9efeb8c0591","Type":"ContainerStarted","Data":"35bdb863b4bfc337eaae843eafe24d67cabc15d59a0c3b8e6a54e8316636381e"}
Sep 16 18:41:55.764761 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:55.764423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67ssr" event={"ID":"a577c313-7fc7-4356-96bc-e9efeb8c0591","Type":"ContainerStarted","Data":"e8e15ab3d83447ffc0dec1957a92378c70080e75bd7f0bf929e086625a7857fd"}
Sep 16 18:41:55.764761 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:55.764440    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67ssr" event={"ID":"a577c313-7fc7-4356-96bc-e9efeb8c0591","Type":"ContainerStarted","Data":"d1889ed3c4a84ca6f20ede2e379ac028fde07fdcc4612138af9efda0106af823"}
Sep 16 18:41:57.772721 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:57.772685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67ssr" event={"ID":"a577c313-7fc7-4356-96bc-e9efeb8c0591","Type":"ContainerStarted","Data":"ccd9d57bfe9b9ab09bd413334a22d7d548031f862842349c2ff90f61c62dbda1"}
Sep 16 18:41:57.806054 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:41:57.806001    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-67ssr" podStartSLOduration=18.209084407 podStartE2EDuration="19.805968397s" totalImagesPullingTime="1.59688399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:41:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:41:57.805150339 +0000 UTC m=+150.173206258" watchObservedRunningTime="2026-09-16 18:41:57.805968397 +0000 UTC m=+150.174024297"
Sep 16 18:42:04.452474 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:42:04.452427    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-675b454989-mb9cg" podUID="d7f06edd-789b-43ab-91d3-e83bb86a928d"
Sep 16 18:42:04.529267 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:42:04.529214    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-4tzv7" podUID="3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93"
Sep 16 18:42:04.674525 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:04.674488    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:04.793521 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:04.793488    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:42:04.793719 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:04.793487    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:42:04.825282 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:04.825244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:06.253542 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:42:06.253502    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-b44bb" podUID="85cfc9e7-4f44-40c4-8080-daa1472cf544"
Sep 16 18:42:06.921689 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:06.921650    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.019377 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:07.019336    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.199443 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:07.199366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.374703 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:07.374666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.637705 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:07.637676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:07.760221 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:07.760147    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:09.394739 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.394690    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:42:09.397264 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.397240    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"image-registry-675b454989-mb9cg\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") " pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:42:09.495099 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.495057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:42:09.497324 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.497291    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93-metrics-tls\") pod \"dns-default-4tzv7\" (UID: \"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93\") " pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:42:09.596990 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.596951    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-24cbm\""
Sep 16 18:42:09.600751 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.600735    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tm7mf\""
Sep 16 18:42:09.604046 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.604029    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:42:09.604135 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.604116    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:42:09.904472 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.904396    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-675b454989-mb9cg"]
Sep 16 18:42:09.907340 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:42:09.907273    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f06edd_789b_43ab_91d3_e83bb86a928d.slice/crio-bcbc593027f965c53fad9602c94970e11375f2999af65a950f4e633725dcfa9f WatchSource:0}: Error finding container bcbc593027f965c53fad9602c94970e11375f2999af65a950f4e633725dcfa9f: Status 404 returned error can't find the container with id bcbc593027f965c53fad9602c94970e11375f2999af65a950f4e633725dcfa9f
Sep 16 18:42:09.908102 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:09.908082    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4tzv7"]
Sep 16 18:42:09.909032 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:42:09.909012    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc3f93a_7244_4ef2_80c6_ba4af3ce9d93.slice/crio-b41d19a32fc274df545096c3e9958c60a3f5df18782923ceccc59cba797a827e WatchSource:0}: Error finding container b41d19a32fc274df545096c3e9958c60a3f5df18782923ceccc59cba797a827e: Status 404 returned error can't find the container with id b41d19a32fc274df545096c3e9958c60a3f5df18782923ceccc59cba797a827e
Sep 16 18:42:10.813542 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:10.813505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675b454989-mb9cg" event={"ID":"d7f06edd-789b-43ab-91d3-e83bb86a928d","Type":"ContainerStarted","Data":"c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d"}
Sep 16 18:42:10.814030 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:10.813563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675b454989-mb9cg" event={"ID":"d7f06edd-789b-43ab-91d3-e83bb86a928d","Type":"ContainerStarted","Data":"bcbc593027f965c53fad9602c94970e11375f2999af65a950f4e633725dcfa9f"}
Sep 16 18:42:10.814030 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:10.813620    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:42:10.815178 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:10.815112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4tzv7" event={"ID":"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93","Type":"ContainerStarted","Data":"b41d19a32fc274df545096c3e9958c60a3f5df18782923ceccc59cba797a827e"}
Sep 16 18:42:10.845862 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:10.845814    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-675b454989-mb9cg" podStartSLOduration=161.84579756 podStartE2EDuration="2m41.84579756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:10.843048781 +0000 UTC m=+163.211104704" watchObservedRunningTime="2026-09-16 18:42:10.84579756 +0000 UTC m=+163.213853455"
Sep 16 18:42:11.448621 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.448573    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5" podUID="dea3e2ac-1b6a-4571-9c66-8014c23d2710" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused"
Sep 16 18:42:11.772769 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.772671    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz" podUID="641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686" containerName="addon-agent" probeResult="failure" output="Get \"http://10.132.0.7:8000/healthz\": dial tcp 10.132.0.7:8000: connect: connection refused"
Sep 16 18:42:11.781042 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.781008    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5" podUID="dea3e2ac-1b6a-4571-9c66-8014c23d2710" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/healthz\": dial tcp 10.132.0.8:8000: connect: connection refused"
Sep 16 18:42:11.820271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.820240    2559 generic.go:356] "Generic (PLEG): container finished" podID="dea3e2ac-1b6a-4571-9c66-8014c23d2710" containerID="8b667df73006464edeb4be343f75b076c426e5f73d2ece70105219de3641ce30" exitCode=1
Sep 16 18:42:11.820702 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.820313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5" event={"ID":"dea3e2ac-1b6a-4571-9c66-8014c23d2710","Type":"ContainerDied","Data":"8b667df73006464edeb4be343f75b076c426e5f73d2ece70105219de3641ce30"}
Sep 16 18:42:11.820702 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.820672    2559 scope.go:117] "RemoveContainer" containerID="8b667df73006464edeb4be343f75b076c426e5f73d2ece70105219de3641ce30"
Sep 16 18:42:11.821811 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.821788    2559 generic.go:356] "Generic (PLEG): container finished" podID="641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686" containerID="4a3037cbf21b44556990d666b03dc813ac7c4b9518045ac1a19c1431a8bee877" exitCode=255
Sep 16 18:42:11.821920 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.821852    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz" event={"ID":"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686","Type":"ContainerDied","Data":"4a3037cbf21b44556990d666b03dc813ac7c4b9518045ac1a19c1431a8bee877"}
Sep 16 18:42:11.822165 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.822148    2559 scope.go:117] "RemoveContainer" containerID="4a3037cbf21b44556990d666b03dc813ac7c4b9518045ac1a19c1431a8bee877"
Sep 16 18:42:11.823791 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.823771    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4tzv7" event={"ID":"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93","Type":"ContainerStarted","Data":"62257e9204d99c47b00029c1ee251fe85b57693aa33f4a342b8ae92bf7ecbc2a"}
Sep 16 18:42:11.823864 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.823811    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4tzv7" event={"ID":"3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93","Type":"ContainerStarted","Data":"ff55465d656bd803b9836e4236013ed6f09fb72aa23c48ba369ff5adfb27c52f"}
Sep 16 18:42:11.824028 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.824013    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:42:11.894502 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:11.894440    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4tzv7" podStartSLOduration=129.472457819 podStartE2EDuration="2m10.894417236s" totalImagesPullingTime="1.421959417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:11.893579556 +0000 UTC m=+164.261635475" watchObservedRunningTime="2026-09-16 18:42:11.894417236 +0000 UTC m=+164.262473139"
Sep 16 18:42:12.310570 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:12.310528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:12.404674 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:12.404641    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:12.480564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:12.480529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:12.828634 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:12.828597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5" event={"ID":"dea3e2ac-1b6a-4571-9c66-8014c23d2710","Type":"ContainerStarted","Data":"c4ccc7493b7322f0668b2b3135467db2483ceada5a117d274c12c2e530375756"}
Sep 16 18:42:12.829084 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:12.828904    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:42:12.830033 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:12.830010    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d44b6bbcd-fczz5"
Sep 16 18:42:12.830436 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:12.830415    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c4c5c6988-8b5rz" event={"ID":"641e9dd1-e8eb-4f1d-ac46-e3f89d6dd686","Type":"ContainerStarted","Data":"c5d866a404a919533d9545af7a43e63c6dbb296c6113849cd86891368e2e1e63"}
Sep 16 18:42:14.447688 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:14.447647    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:16.627250 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:16.627211    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:16.750232 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:16.750193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:17.129388 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.129359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:17.308271 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.308233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:17.488051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.488013    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:17.512409 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.512379    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xg6dr"]
Sep 16 18:42:17.515390 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.515374    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.528909 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.528892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:42:17.529058 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.529043    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:42:17.530583 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.530566    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:42:17.530857 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.530842    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:42:17.531488 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.531475    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xrql4\""
Sep 16 18:42:17.531488 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.531480    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:42:17.531945 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.531931    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:42:17.647821 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.644797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:17.656619 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656593    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-root\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.656749 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656626    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-accelerators-collector-config\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.656749 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656646    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-textfile\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.656749 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656664    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.656749 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-wtmp\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.656954 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-metrics-client-ca\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.656954 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pcl\" (UniqueName: \"kubernetes.io/projected/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-kube-api-access-97pcl\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.656954 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-tls\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.656954 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.656902    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-sys\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.757841 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-sys\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.757841 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757824    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-root\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-accelerators-collector-config\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-sys\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-textfile\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-wtmp\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757933    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-root\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.757953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-metrics-client-ca\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.758006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pcl\" (UniqueName: \"kubernetes.io/projected/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-kube-api-access-97pcl\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758068 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.758051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-tls\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758382 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.758107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-wtmp\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758435 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.758414    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-textfile\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758639 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.758609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-accelerators-collector-config\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.758758 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.758647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-metrics-client-ca\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.760383 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.760364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-tls\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.760509 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.760491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.775791 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.775756    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pcl\" (UniqueName: \"kubernetes.io/projected/e5541b1b-46c2-4be3-8fc0-3da9cc0e0002-kube-api-access-97pcl\") pod \"node-exporter-xg6dr\" (UID: \"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002\") " pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.823858 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.823824    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xg6dr"
Sep 16 18:42:17.834672 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:42:17.834645    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5541b1b_46c2_4be3_8fc0_3da9cc0e0002.slice/crio-babde0e421563d64dfcef056c2bdea1aa1c78279f87264eba84c46c96d63840c WatchSource:0}: Error finding container babde0e421563d64dfcef056c2bdea1aa1c78279f87264eba84c46c96d63840c: Status 404 returned error can't find the container with id babde0e421563d64dfcef056c2bdea1aa1c78279f87264eba84c46c96d63840c
Sep 16 18:42:17.849383 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.849343    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xg6dr" event={"ID":"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002","Type":"ContainerStarted","Data":"babde0e421563d64dfcef056c2bdea1aa1c78279f87264eba84c46c96d63840c"}
Sep 16 18:42:17.936835 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:17.936799    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:18.106087 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:18.105992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:18.219319 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:18.219285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:18.221855 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:18.221838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:42:18.388230 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:18.388139    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:18.525286 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:18.525253    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:19.858410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:19.858374    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5541b1b-46c2-4be3-8fc0-3da9cc0e0002" containerID="c3d43cf21e7e14c9b2274ed2c906bbfca15ee4d759cad1e89a3db3e01aa3da0e" exitCode=0
Sep 16 18:42:19.858775 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:19.858427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xg6dr" event={"ID":"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002","Type":"ContainerDied","Data":"c3d43cf21e7e14c9b2274ed2c906bbfca15ee4d759cad1e89a3db3e01aa3da0e"}
Sep 16 18:42:20.863665 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:20.863628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xg6dr" event={"ID":"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002","Type":"ContainerStarted","Data":"7f247d4d2117ab10ec4e9d064438cb97a5516f3f3608f48219e8f5979a2e2257"}
Sep 16 18:42:20.864056 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:20.863687    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xg6dr" event={"ID":"e5541b1b-46c2-4be3-8fc0-3da9cc0e0002","Type":"ContainerStarted","Data":"2da66034723847f961b2c9b01ded911216899cc8be22777e4b7f283fcd85e2ce"}
Sep 16 18:42:20.888779 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:20.888733    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xg6dr" podStartSLOduration=2.802570493 podStartE2EDuration="3.888718798s" totalImagesPullingTime="1.024423983s" totalInitContainerRuntime="61.724322ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:20.887225948 +0000 UTC m=+173.255281871" watchObservedRunningTime="2026-09-16 18:42:20.888718798 +0000 UTC m=+173.256774695"
Sep 16 18:42:21.601349 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:21.601316    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:21.768749 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:21.768715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:21.832658 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:21.832623    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4tzv7"
Sep 16 18:42:26.549290 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:26.549260    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675b454989-mb9cg"]
Sep 16 18:42:26.553734 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:26.553708    2559 patch_prober.go:28] interesting pod/image-registry-675b454989-mb9cg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:42:26.553867 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:26.553750    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-675b454989-mb9cg" podUID="d7f06edd-789b-43ab-91d3-e83bb86a928d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:42:26.695735 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:26.695700    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:26.773765 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:26.773727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:29.755828 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:29.755786    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:29.883228 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:29.883166    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:30.028455 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:30.028348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:30.233706 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:30.233673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:30.320453 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:30.320371    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:31.039721 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:31.039684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:31.121506 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:31.121477    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:31.440905 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:31.440817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:31.630266 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:31.630227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:32.849537 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:32.849499    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:32.919189 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:32.919141    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:33.526678 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:33.526646    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:33.594853 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:33.594818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:36.185881 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:36.185834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:36.260189 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:36.260158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:36.553918 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:36.553893    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:42:40.037997 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:40.037946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:40.139077 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:40.139035    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:41.798222 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:41.798180    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" podUID="a2d4b9c7-33e3-4647-938c-541b1b2b9648" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:42:42.613025 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:42.612995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:42.696739 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:42.696704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:45.134488 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:45.134452    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:45.229260 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:45.229228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:51.424961 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.424916    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:51.491996 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.491946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:51.567956 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.567902    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-675b454989-mb9cg" podUID="d7f06edd-789b-43ab-91d3-e83bb86a928d" containerName="registry" containerID="cri-o://c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d" gracePeriod=30
Sep 16 18:42:51.797465 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.797436    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:42:51.797987 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.797946    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" podUID="a2d4b9c7-33e3-4647-938c-541b1b2b9648" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:42:51.805127 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805109    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-certificates\") pod \"d7f06edd-789b-43ab-91d3-e83bb86a928d\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") "
Sep 16 18:42:51.805211 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805136    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-image-registry-private-configuration\") pod \"d7f06edd-789b-43ab-91d3-e83bb86a928d\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") "
Sep 16 18:42:51.805211 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805163    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7f06edd-789b-43ab-91d3-e83bb86a928d-ca-trust-extracted\") pod \"d7f06edd-789b-43ab-91d3-e83bb86a928d\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") "
Sep 16 18:42:51.805287 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805216    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-trusted-ca\") pod \"d7f06edd-789b-43ab-91d3-e83bb86a928d\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") "
Sep 16 18:42:51.805287 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805248    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-installation-pull-secrets\") pod \"d7f06edd-789b-43ab-91d3-e83bb86a928d\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") "
Sep 16 18:42:51.805457 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805287    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") pod \"d7f06edd-789b-43ab-91d3-e83bb86a928d\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") "
Sep 16 18:42:51.805457 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805314    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4jcv\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-kube-api-access-x4jcv\") pod \"d7f06edd-789b-43ab-91d3-e83bb86a928d\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") "
Sep 16 18:42:51.805457 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805339    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-bound-sa-token\") pod \"d7f06edd-789b-43ab-91d3-e83bb86a928d\" (UID: \"d7f06edd-789b-43ab-91d3-e83bb86a928d\") "
Sep 16 18:42:51.805615 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805546    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d7f06edd-789b-43ab-91d3-e83bb86a928d" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:51.805721 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.805699    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d7f06edd-789b-43ab-91d3-e83bb86a928d" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:51.807715 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.807668    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d7f06edd-789b-43ab-91d3-e83bb86a928d" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:51.807997 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.807939    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d7f06edd-789b-43ab-91d3-e83bb86a928d" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:51.808091 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.808014    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d7f06edd-789b-43ab-91d3-e83bb86a928d" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:51.808156 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.808131    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d7f06edd-789b-43ab-91d3-e83bb86a928d" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:51.808195 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.808147    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-kube-api-access-x4jcv" (OuterVolumeSpecName: "kube-api-access-x4jcv") pod "d7f06edd-789b-43ab-91d3-e83bb86a928d" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d"). InnerVolumeSpecName "kube-api-access-x4jcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:51.814075 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.814054    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f06edd-789b-43ab-91d3-e83bb86a928d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d7f06edd-789b-43ab-91d3-e83bb86a928d" (UID: "d7f06edd-789b-43ab-91d3-e83bb86a928d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:51.906782 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.906751    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-trusted-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:42:51.906782 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.906776    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-installation-pull-secrets\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:42:51.906994 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.906785    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-tls\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:42:51.906994 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.906802    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4jcv\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-kube-api-access-x4jcv\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:42:51.906994 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.906812    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7f06edd-789b-43ab-91d3-e83bb86a928d-bound-sa-token\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:42:51.906994 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.906820    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7f06edd-789b-43ab-91d3-e83bb86a928d-registry-certificates\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:42:51.906994 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.906828    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7f06edd-789b-43ab-91d3-e83bb86a928d-image-registry-private-configuration\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:42:51.906994 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.906836    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7f06edd-789b-43ab-91d3-e83bb86a928d-ca-trust-extracted\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:42:51.961748 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.961715    2559 generic.go:356] "Generic (PLEG): container finished" podID="d7f06edd-789b-43ab-91d3-e83bb86a928d" containerID="c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d" exitCode=0
Sep 16 18:42:51.961881 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.961758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675b454989-mb9cg" event={"ID":"d7f06edd-789b-43ab-91d3-e83bb86a928d","Type":"ContainerDied","Data":"c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d"}
Sep 16 18:42:51.961881 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.961766    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-675b454989-mb9cg"
Sep 16 18:42:51.961881 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.961806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-675b454989-mb9cg" event={"ID":"d7f06edd-789b-43ab-91d3-e83bb86a928d","Type":"ContainerDied","Data":"bcbc593027f965c53fad9602c94970e11375f2999af65a950f4e633725dcfa9f"}
Sep 16 18:42:51.962591 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.962568    2559 scope.go:117] "RemoveContainer" containerID="c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d"
Sep 16 18:42:51.971349 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.971333    2559 scope.go:117] "RemoveContainer" containerID="c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d"
Sep 16 18:42:51.971628 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:42:51.971607    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d\": container with ID starting with c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d not found: ID does not exist" containerID="c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d"
Sep 16 18:42:51.971670 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.971636    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d"} err="failed to get container status \"c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d\": rpc error: code = NotFound desc = could not find container \"c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d\": container with ID starting with c1b8075f9bf49250733d5a98c9c9b9b0d6819678c3909f02d75ccb282903b04d not found: ID does not exist"
Sep 16 18:42:51.983681 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.983656    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-675b454989-mb9cg"]
Sep 16 18:42:51.993132 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:51.993109    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-675b454989-mb9cg"]
Sep 16 18:42:52.221421 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:52.221349    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7f06edd-789b-43ab-91d3-e83bb86a928d" path="/var/lib/kubelet/pods/d7f06edd-789b-43ab-91d3-e83bb86a928d/volumes"
Sep 16 18:42:54.656140 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:54.656106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:54.760153 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:54.760122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:58.639168 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:58.639137    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xqbwk_d3342a0b-c9c9-46bc-beab-8fd7574ef11b/serve-healthcheck-canary/0.log"
Sep 16 18:42:58.858259 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:58.858227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:58.935501 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:58.935417    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:42:59.516498 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:42:59.516456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:01.797372 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:01.797335    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" podUID="a2d4b9c7-33e3-4647-938c-541b1b2b9648" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 18:43:01.797796 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:01.797399    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6"
Sep 16 18:43:01.797844 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:01.797828    2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6756d1d58d9585e125bb6f213d9e52f7832b0d668d6947840c094aa87ed053ca"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 18:43:01.797904 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:01.797862    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" podUID="a2d4b9c7-33e3-4647-938c-541b1b2b9648" containerName="service-proxy" containerID="cri-o://6756d1d58d9585e125bb6f213d9e52f7832b0d668d6947840c094aa87ed053ca" gracePeriod=30
Sep 16 18:43:01.999955 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:01.999920    2559 generic.go:356] "Generic (PLEG): container finished" podID="a2d4b9c7-33e3-4647-938c-541b1b2b9648" containerID="6756d1d58d9585e125bb6f213d9e52f7832b0d668d6947840c094aa87ed053ca" exitCode=2
Sep 16 18:43:02.000131 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:02.000006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" event={"ID":"a2d4b9c7-33e3-4647-938c-541b1b2b9648","Type":"ContainerDied","Data":"6756d1d58d9585e125bb6f213d9e52f7832b0d668d6947840c094aa87ed053ca"}
Sep 16 18:43:02.000131 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:02.000050    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7975f87cd4-qtpm6" event={"ID":"a2d4b9c7-33e3-4647-938c-541b1b2b9648","Type":"ContainerStarted","Data":"b12afb30fb72b5d32f17dd0a77fd2ef330a967244e63fbe2173894487e345dd8"}
Sep 16 18:43:03.935446 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:03.935412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:04.036772 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:04.036736    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:04.119265 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:04.119232    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:06.456063 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:06.456028    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:06.590390 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:06.590355    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:30.708201 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:30.708164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:39.856756 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:39.856715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:43:39.858954 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:39.858930    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85cfc9e7-4f44-40c4-8080-daa1472cf544-metrics-certs\") pod \"network-metrics-daemon-b44bb\" (UID: \"85cfc9e7-4f44-40c4-8080-daa1472cf544\") " pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:43:40.125287 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:40.125207    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l6kxj\""
Sep 16 18:43:40.132657 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:40.132637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b44bb"
Sep 16 18:43:40.404888 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:40.404865    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b44bb"]
Sep 16 18:43:40.407283 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:43:40.407255    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cfc9e7_4f44_40c4_8080_daa1472cf544.slice/crio-fd1c658417005130cf0a8ea496d2fa917828d16495e5a8d3e93b3be0dfea0102 WatchSource:0}: Error finding container fd1c658417005130cf0a8ea496d2fa917828d16495e5a8d3e93b3be0dfea0102: Status 404 returned error can't find the container with id fd1c658417005130cf0a8ea496d2fa917828d16495e5a8d3e93b3be0dfea0102
Sep 16 18:43:41.128112 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:41.128078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b44bb" event={"ID":"85cfc9e7-4f44-40c4-8080-daa1472cf544","Type":"ContainerStarted","Data":"fd1c658417005130cf0a8ea496d2fa917828d16495e5a8d3e93b3be0dfea0102"}
Sep 16 18:43:42.132894 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:42.132861    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b44bb" event={"ID":"85cfc9e7-4f44-40c4-8080-daa1472cf544","Type":"ContainerStarted","Data":"7770c732e7ee8e9a795297fcbe9cce013267401f69a6bc92dbf6626a10fec7c3"}
Sep 16 18:43:42.133275 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:42.132912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b44bb" event={"ID":"85cfc9e7-4f44-40c4-8080-daa1472cf544","Type":"ContainerStarted","Data":"6c9617d7151ec76b7a9f4d4ff806c1d66b9e848bf0a15087bb14a0218b75492e"}
Sep 16 18:43:42.151340 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:42.151297    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b44bb" podStartSLOduration=252.150751031 podStartE2EDuration="4m13.151282593s" totalImagesPullingTime="1.000531562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:39:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:42.149328321 +0000 UTC m=+254.517384245" watchObservedRunningTime="2026-09-16 18:43:42.151282593 +0000 UTC m=+254.519338494"
Sep 16 18:43:42.259505 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:42.259471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:42.349355 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:42.349318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:45.965824 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:45.965776    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:46.066408 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:46.066369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:46.767298 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:46.767263    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:46.864456 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:43:46.864425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:27.964180 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:44:27.964149    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:44:27.967239 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:44:27.967217    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:44:27.971198 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:44:27.971175    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:44:29.020182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:44:29.020150    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:29.096987 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:44:29.096935    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:33.234613 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:45:33.234518    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:49.658609 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:45:49.658564    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:25.366920 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:47:25.366885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:26.653554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:47:26.653515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:28.623017 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:47:28.622984    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:38.513996 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:48:38.513890    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:38.593118 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:48:38.593091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:41.342446 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:48:41.342412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:50.457045 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:48:50.457011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:50.526439 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:48:50.526410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:53.964514 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:48:53.964479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:05.225813 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:05.225773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:05.330610 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:05.330580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:05.412042 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:05.412010    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:06.425098 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:06.425066    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:06.526923 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:06.526888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:27.988118 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:27.988091    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:49:27.988575 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:27.988332    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:49:43.487624 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:43.487586    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:43.585940 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:49:43.585900    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:10.411558 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:10.411483    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:11.313776 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:11.313741    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:13.910442 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:13.910407    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:14.010309 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:14.010268    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:15.636270 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:15.636239    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:15.771038 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:15.771006    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:15.870660 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:15.870631    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:18.043853 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:18.043817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:18.444106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:18.444020    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:20.891017 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:20.890965    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:20.954355 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:20.954319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:26.254935 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:26.254903    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:26.898638 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:26.898597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:26.995622 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:26.995584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:28.364920 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:28.364890    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:28.452203 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:28.452172    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:33.188821 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:33.188784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:33.296505 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:33.296473    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:34.866515 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:34.866482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:39.091376 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:39.091340    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:53.329453 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:53.329419    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:53.427601 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:53.427570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:53.725427 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:53.725399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.048514 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.048432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.139079 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.139029    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.181884 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.181860    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"]
Sep 16 18:50:54.182099 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.182086    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f06edd-789b-43ab-91d3-e83bb86a928d" containerName="registry"
Sep 16 18:50:54.182144 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.182100    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f06edd-789b-43ab-91d3-e83bb86a928d" containerName="registry"
Sep 16 18:50:54.182177 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.182147    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7f06edd-789b-43ab-91d3-e83bb86a928d" containerName="registry"
Sep 16 18:50:54.184827 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.184813    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.187630 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.187610    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 18:50:54.187843 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.187829    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 18:50:54.188182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.188163    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 18:50:54.188292 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.188182    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gbmdk\""
Sep 16 18:50:54.188662 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.188648    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 18:50:54.203026 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.203003    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"]
Sep 16 18:50:54.239748 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.239719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-tmp\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.239877 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.239754    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-manifests\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.239877 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.239772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.239877 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.239792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-webhook-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.240002 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.239874    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvqg\" (UniqueName: \"kubernetes.io/projected/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-kube-api-access-vzvqg\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.340711 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.340630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-webhook-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.341144 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.340725    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvqg\" (UniqueName: \"kubernetes.io/projected/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-kube-api-access-vzvqg\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.341144 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.340760    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-tmp\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.341144 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.340794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-manifests\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.341144 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.340911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.341316 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.341193    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-manifests\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.343032 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.343013    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-tmp\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.343291 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.343274    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-webhook-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.343361 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.343340    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.351688 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.351661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvqg\" (UniqueName: \"kubernetes.io/projected/dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff-kube-api-access-vzvqg\") pod \"opendatahub-operator-controller-manager-85745479c7-qpspw\" (UID: \"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff\") " pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.353149 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.353113    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.462441 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.462413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.494364 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.494334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:50:54.564385 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.564360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.663601 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.663535    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.796248 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.796218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.838340 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:50:54.838315    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3ca9f7_c181_4ef3_abb4_6cdf27ee7fff.slice/crio-5cbccd184de1da39bb37557bfff33c45e1331ea3f5df2f456531c79c24c34cf7 WatchSource:0}: Error finding container 5cbccd184de1da39bb37557bfff33c45e1331ea3f5df2f456531c79c24c34cf7: Status 404 returned error can't find the container with id 5cbccd184de1da39bb37557bfff33c45e1331ea3f5df2f456531c79c24c34cf7
Sep 16 18:50:54.840295 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.840274    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:54.849895 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.849871    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"]
Sep 16 18:50:54.913099 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.913063    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:54.995463 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:54.995437    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:55.097905 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:55.097880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:55.500749 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:55.500634    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw" event={"ID":"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff","Type":"ContainerStarted","Data":"5cbccd184de1da39bb37557bfff33c45e1331ea3f5df2f456531c79c24c34cf7"}
Sep 16 18:50:58.513459 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:58.513423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw" event={"ID":"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff","Type":"ContainerStarted","Data":"9cac8c6cadac2557f2566cfab7ee199865aeb5b1fc4f33a52b6c84ccdbd14eba"}
Sep 16 18:50:59.148727 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:59.148691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:59.253336 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:59.253304    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:59.518096 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:59.518064    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff" containerID="9cac8c6cadac2557f2566cfab7ee199865aeb5b1fc4f33a52b6c84ccdbd14eba" exitCode=0
Sep 16 18:50:59.518451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:50:59.518112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw" event={"ID":"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff","Type":"ContainerDied","Data":"9cac8c6cadac2557f2566cfab7ee199865aeb5b1fc4f33a52b6c84ccdbd14eba"}
Sep 16 18:51:00.523115 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:00.523079    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw" event={"ID":"dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff","Type":"ContainerStarted","Data":"60b45f1b748a6872061c71929e1bde7f0cb73d600a9d47c9ed2f40d3186f7138"}
Sep 16 18:51:00.523495 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:00.523211    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:51:00.550411 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:00.550363    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw" podStartSLOduration=2.017650569 podStartE2EDuration="6.550346761s" totalImagesPullingTime="3.524477028s" totalInitContainerRuntime="1.008219164s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:00.548822444 +0000 UTC m=+692.916878354" watchObservedRunningTime="2026-09-16 18:51:00.550346761 +0000 UTC m=+692.918402661"
Sep 16 18:51:11.528735 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:11.528699    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85745479c7-qpspw"
Sep 16 18:51:13.202090 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:13.202049    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:13.338010 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:13.337957    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:24.197151 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.197114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:24.233372 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.233343    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"]
Sep 16 18:51:24.236054 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.236039    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.241641 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.241614    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 18:51:24.241790 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.241655    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 18:51:24.241790 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.241691    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:51:24.241790 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.241715    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:51:24.253766 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.253743    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"]
Sep 16 18:51:24.360578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.360539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/274a44e2-960f-4baf-9519-d0106a12856f-tmp\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.360578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.360580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw2nm\" (UniqueName: \"kubernetes.io/projected/274a44e2-960f-4baf-9519-d0106a12856f-kube-api-access-pw2nm\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.360820 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.360682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274a44e2-960f-4baf-9519-d0106a12856f-tls-certs\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.393235 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.393208    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8mvw"]
Sep 16 18:51:24.396185 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.396170    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:24.400112 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.400091    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8rqhq\""
Sep 16 18:51:24.400706 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.400694    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:51:24.416992 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.416943    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8mvw"]
Sep 16 18:51:24.440412 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.440390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:24.461300 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.461273    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274a44e2-960f-4baf-9519-d0106a12856f-tls-certs\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.461453 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.461330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/274a44e2-960f-4baf-9519-d0106a12856f-tmp\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.461453 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.461361    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw2nm\" (UniqueName: \"kubernetes.io/projected/274a44e2-960f-4baf-9519-d0106a12856f-kube-api-access-pw2nm\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.461453 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:51:24.461420    2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 18:51:24.461610 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:51:24.461505    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/274a44e2-960f-4baf-9519-d0106a12856f-tls-certs podName:274a44e2-960f-4baf-9519-d0106a12856f nodeName:}" failed. No retries permitted until 2026-09-16 18:51:24.961488767 +0000 UTC m=+717.329544647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/274a44e2-960f-4baf-9519-d0106a12856f-tls-certs") pod "kube-auth-proxy-7d78bff8dd-sp4kx" (UID: "274a44e2-960f-4baf-9519-d0106a12856f") : secret "kube-auth-proxy-tls" not found
Sep 16 18:51:24.463462 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.463440    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/274a44e2-960f-4baf-9519-d0106a12856f-tmp\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.475339 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.475313    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw2nm\" (UniqueName: \"kubernetes.io/projected/274a44e2-960f-4baf-9519-d0106a12856f-kube-api-access-pw2nm\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.538003 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.537953    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:24.562327 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.562298    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca15ef45-43c9-46f2-916e-2f20a3b61184-cert\") pod \"odh-model-controller-7998c67dc4-p8mvw\" (UID: \"ca15ef45-43c9-46f2-916e-2f20a3b61184\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:24.562437 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.562337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xplcf\" (UniqueName: \"kubernetes.io/projected/ca15ef45-43c9-46f2-916e-2f20a3b61184-kube-api-access-xplcf\") pod \"odh-model-controller-7998c67dc4-p8mvw\" (UID: \"ca15ef45-43c9-46f2-916e-2f20a3b61184\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:24.648459 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.648423    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:24.663607 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.663574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca15ef45-43c9-46f2-916e-2f20a3b61184-cert\") pod \"odh-model-controller-7998c67dc4-p8mvw\" (UID: \"ca15ef45-43c9-46f2-916e-2f20a3b61184\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:24.663733 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.663619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xplcf\" (UniqueName: \"kubernetes.io/projected/ca15ef45-43c9-46f2-916e-2f20a3b61184-kube-api-access-xplcf\") pod \"odh-model-controller-7998c67dc4-p8mvw\" (UID: \"ca15ef45-43c9-46f2-916e-2f20a3b61184\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:24.665918 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.665891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ca15ef45-43c9-46f2-916e-2f20a3b61184-cert\") pod \"odh-model-controller-7998c67dc4-p8mvw\" (UID: \"ca15ef45-43c9-46f2-916e-2f20a3b61184\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:24.680489 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.680462    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xplcf\" (UniqueName: \"kubernetes.io/projected/ca15ef45-43c9-46f2-916e-2f20a3b61184-kube-api-access-xplcf\") pod \"odh-model-controller-7998c67dc4-p8mvw\" (UID: \"ca15ef45-43c9-46f2-916e-2f20a3b61184\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:24.705401 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.705375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:24.745516 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.745490    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:24.848564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.848532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:24.965926 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.965853    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274a44e2-960f-4baf-9519-d0106a12856f-tls-certs\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.968107 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.968087    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274a44e2-960f-4baf-9519-d0106a12856f-tls-certs\") pod \"kube-auth-proxy-7d78bff8dd-sp4kx\" (UID: \"274a44e2-960f-4baf-9519-d0106a12856f\") " pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:24.980054 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:24.980028    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8mvw"]
Sep 16 18:51:24.982735 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:51:24.982711    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca15ef45_43c9_46f2_916e_2f20a3b61184.slice/crio-bdf945f32934f58e744e4e7c9270146b4c92f2528ef25ae4beb890125b1c8355 WatchSource:0}: Error finding container bdf945f32934f58e744e4e7c9270146b4c92f2528ef25ae4beb890125b1c8355: Status 404 returned error can't find the container with id bdf945f32934f58e744e4e7c9270146b4c92f2528ef25ae4beb890125b1c8355
Sep 16 18:51:25.144916 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.144876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"
Sep 16 18:51:25.430174 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:51:25.430146    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274a44e2_960f_4baf_9519_d0106a12856f.slice/crio-dbf258f30c5cdf4f37353354cccc54134fa56329e3dfd33d169b1d8115cd5cc8 WatchSource:0}: Error finding container dbf258f30c5cdf4f37353354cccc54134fa56329e3dfd33d169b1d8115cd5cc8: Status 404 returned error can't find the container with id dbf258f30c5cdf4f37353354cccc54134fa56329e3dfd33d169b1d8115cd5cc8
Sep 16 18:51:25.433130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.433107    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx"]
Sep 16 18:51:25.518342 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.518310    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7kv5l"]
Sep 16 18:51:25.521426 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.521405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:25.523267 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.523235    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:25.526717 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.526696    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 18:51:25.543569 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.543545    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7kv5l"]
Sep 16 18:51:25.612454 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.612419    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx" event={"ID":"274a44e2-960f-4baf-9519-d0106a12856f","Type":"ContainerStarted","Data":"dbf258f30c5cdf4f37353354cccc54134fa56329e3dfd33d169b1d8115cd5cc8"}
Sep 16 18:51:25.613770 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.613740    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw" event={"ID":"ca15ef45-43c9-46f2-916e-2f20a3b61184","Type":"ContainerStarted","Data":"bdf945f32934f58e744e4e7c9270146b4c92f2528ef25ae4beb890125b1c8355"}
Sep 16 18:51:25.670199 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.670161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjs4m\" (UniqueName: \"kubernetes.io/projected/049a0535-4721-45af-9da0-04e3ff380856-kube-api-access-qjs4m\") pod \"model-serving-api-54d89bd5fc-7kv5l\" (UID: \"049a0535-4721-45af-9da0-04e3ff380856\") " pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:25.670361 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.670230    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049a0535-4721-45af-9da0-04e3ff380856-tls-certs\") pod \"model-serving-api-54d89bd5fc-7kv5l\" (UID: \"049a0535-4721-45af-9da0-04e3ff380856\") " pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:25.771521 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.771482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049a0535-4721-45af-9da0-04e3ff380856-tls-certs\") pod \"model-serving-api-54d89bd5fc-7kv5l\" (UID: \"049a0535-4721-45af-9da0-04e3ff380856\") " pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:25.771723 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.771585    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjs4m\" (UniqueName: \"kubernetes.io/projected/049a0535-4721-45af-9da0-04e3ff380856-kube-api-access-qjs4m\") pod \"model-serving-api-54d89bd5fc-7kv5l\" (UID: \"049a0535-4721-45af-9da0-04e3ff380856\") " pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:25.771723 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:51:25.771663    2559 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 18:51:25.771834 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:51:25.771741    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049a0535-4721-45af-9da0-04e3ff380856-tls-certs podName:049a0535-4721-45af-9da0-04e3ff380856 nodeName:}" failed. No retries permitted until 2026-09-16 18:51:26.271724892 +0000 UTC m=+718.639780787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/049a0535-4721-45af-9da0-04e3ff380856-tls-certs") pod "model-serving-api-54d89bd5fc-7kv5l" (UID: "049a0535-4721-45af-9da0-04e3ff380856") : secret "model-serving-api-tls" not found
Sep 16 18:51:25.787588 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.787535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjs4m\" (UniqueName: \"kubernetes.io/projected/049a0535-4721-45af-9da0-04e3ff380856-kube-api-access-qjs4m\") pod \"model-serving-api-54d89bd5fc-7kv5l\" (UID: \"049a0535-4721-45af-9da0-04e3ff380856\") " pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:25.914493 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:25.914459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:26.067111 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.067031    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:26.081353 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.081320    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"]
Sep 16 18:51:26.084585 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.084557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.089127 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.089098    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rzjpm\""
Sep 16 18:51:26.089232 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.089160    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 18:51:26.089363 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.089348    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 18:51:26.089892 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.089872    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:51:26.090418 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.090397    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 18:51:26.112318 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.112287    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"]
Sep 16 18:51:26.172489 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.172409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:26.176043 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.175851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.176043 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.175922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5193dacd-56cd-42d7-8284-73da8826fbf4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.176043 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.175998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpmn\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-kube-api-access-wvpmn\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.176043 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.176036    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.176380 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.176063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.176380 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.176103    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.176380 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.176136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.277104 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049a0535-4721-45af-9da0-04e3ff380856-tls-certs\") pod \"model-serving-api-54d89bd5fc-7kv5l\" (UID: \"049a0535-4721-45af-9da0-04e3ff380856\") " pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:26.277279 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277116    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5193dacd-56cd-42d7-8284-73da8826fbf4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.277279 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277159    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpmn\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-kube-api-access-wvpmn\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.277279 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277194    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.277279 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.277279 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.277537 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.277537 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.278004 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.277962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.288430 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.288401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/049a0535-4721-45af-9da0-04e3ff380856-tls-certs\") pod \"model-serving-api-54d89bd5fc-7kv5l\" (UID: \"049a0535-4721-45af-9da0-04e3ff380856\") " pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:26.288868 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.288841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5193dacd-56cd-42d7-8284-73da8826fbf4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.292816 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.292793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.293656 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.293611    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.294533 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.294493    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.294632 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.294582    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.297235 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.297213    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpmn\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-kube-api-access-wvpmn\") pod \"istiod-openshift-gateway-7b7995d5cd-czq7n\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.398088 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.397828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:26.432894 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:26.432862    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:27.420711 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:27.420676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:29.046863 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.046646    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 18:51:29.259756 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.259730    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"]
Sep 16 18:51:29.261383 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:51:29.261353    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5193dacd_56cd_42d7_8284_73da8826fbf4.slice/crio-fa64412ed169e1874c6accdd3378f09fc3fc8b8da309a8a9d067bd27069c4a1d WatchSource:0}: Error finding container fa64412ed169e1874c6accdd3378f09fc3fc8b8da309a8a9d067bd27069c4a1d: Status 404 returned error can't find the container with id fa64412ed169e1874c6accdd3378f09fc3fc8b8da309a8a9d067bd27069c4a1d
Sep 16 18:51:29.263866 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.263835    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7kv5l"]
Sep 16 18:51:29.266429 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:51:29.266406    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049a0535_4721_45af_9da0_04e3ff380856.slice/crio-3a4a035477cea9fe82682d8c698c2c0c8d6db4bc4515ca494be4c16c5493a4a8 WatchSource:0}: Error finding container 3a4a035477cea9fe82682d8c698c2c0c8d6db4bc4515ca494be4c16c5493a4a8: Status 404 returned error can't find the container with id 3a4a035477cea9fe82682d8c698c2c0c8d6db4bc4515ca494be4c16c5493a4a8
Sep 16 18:51:29.634776 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.634685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw" event={"ID":"ca15ef45-43c9-46f2-916e-2f20a3b61184","Type":"ContainerStarted","Data":"7777320a9d71ba46d3d8f8705261cdf2a52ef2700d1ff9051accb03efc0cf9e2"}
Sep 16 18:51:29.635026 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.634800    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:29.636202 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.636169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n" event={"ID":"5193dacd-56cd-42d7-8284-73da8826fbf4","Type":"ContainerStarted","Data":"fa64412ed169e1874c6accdd3378f09fc3fc8b8da309a8a9d067bd27069c4a1d"}
Sep 16 18:51:29.637795 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.637776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx" event={"ID":"274a44e2-960f-4baf-9519-d0106a12856f","Type":"ContainerStarted","Data":"cd037335f004cb01fbfcc3a2721bc0b5e4703fadfb8c1c309125235dea954c53"}
Sep 16 18:51:29.639143 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.639116    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l" event={"ID":"049a0535-4721-45af-9da0-04e3ff380856","Type":"ContainerStarted","Data":"3a4a035477cea9fe82682d8c698c2c0c8d6db4bc4515ca494be4c16c5493a4a8"}
Sep 16 18:51:29.655630 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.655580    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw" podStartSLOduration=1.6452491299999998 podStartE2EDuration="5.655563852s" totalImagesPullingTime="4.010314722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:29.654715167 +0000 UTC m=+722.022771081" watchObservedRunningTime="2026-09-16 18:51:29.655563852 +0000 UTC m=+722.023619759"
Sep 16 18:51:29.674087 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:29.674044    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d78bff8dd-sp4kx" podStartSLOduration=2.064579251 podStartE2EDuration="5.674032494s" totalImagesPullingTime="3.609453243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:29.672728337 +0000 UTC m=+722.040784259" watchObservedRunningTime="2026-09-16 18:51:29.674032494 +0000 UTC m=+722.042088402"
Sep 16 18:51:31.647842 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:31.647811    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l" event={"ID":"049a0535-4721-45af-9da0-04e3ff380856","Type":"ContainerStarted","Data":"af96a116b39942c79549f55a74be8fe5224b8d8d6e6020ddff7a314cd18aadc7"}
Sep 16 18:51:31.648241 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:31.647947    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:31.706842 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:31.706769    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l" podStartSLOduration=4.810893134 podStartE2EDuration="6.706746161s" totalImagesPullingTime="1.895853027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:31.703756073 +0000 UTC m=+724.071811993" watchObservedRunningTime="2026-09-16 18:51:31.706746161 +0000 UTC m=+724.074802064"
Sep 16 18:51:32.915354 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:32.915318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:33.812578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.812543    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"]
Sep 16 18:51:33.815669 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.815645    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:33.820159 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.820139    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 18:51:33.822297 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.822269    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z74zj\""
Sep 16 18:51:33.822381 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.822322    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 18:51:33.822560 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.822543    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 18:51:33.822638 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.822628    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 18:51:33.822693 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.822638    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:51:33.838372 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.838350    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"]
Sep 16 18:51:33.935934 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.935899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d35ced6-b776-4ff3-981c-fc3679d0fca2-manager-config\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:33.935934 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.935938    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d35ced6-b776-4ff3-981c-fc3679d0fca2-cert\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:33.936339 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.936042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d35ced6-b776-4ff3-981c-fc3679d0fca2-metrics-cert\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:33.936339 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:33.936084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp6j2\" (UniqueName: \"kubernetes.io/projected/1d35ced6-b776-4ff3-981c-fc3679d0fca2-kube-api-access-wp6j2\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.037166 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.037132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d35ced6-b776-4ff3-981c-fc3679d0fca2-cert\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.037363 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.037193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d35ced6-b776-4ff3-981c-fc3679d0fca2-metrics-cert\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.037363 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.037223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp6j2\" (UniqueName: \"kubernetes.io/projected/1d35ced6-b776-4ff3-981c-fc3679d0fca2-kube-api-access-wp6j2\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.037363 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.037250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d35ced6-b776-4ff3-981c-fc3679d0fca2-manager-config\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.037905 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.037885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1d35ced6-b776-4ff3-981c-fc3679d0fca2-manager-config\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.039725 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.039699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d35ced6-b776-4ff3-981c-fc3679d0fca2-cert\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.039812 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.039769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1d35ced6-b776-4ff3-981c-fc3679d0fca2-metrics-cert\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.058059 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.058036    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp6j2\" (UniqueName: \"kubernetes.io/projected/1d35ced6-b776-4ff3-981c-fc3679d0fca2-kube-api-access-wp6j2\") pod \"lws-controller-manager-5f86d45999-zx52x\" (UID: \"1d35ced6-b776-4ff3-981c-fc3679d0fca2\") " pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.125020 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.124929    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:34.412132 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.412058    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"]
Sep 16 18:51:34.414821 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:51:34.414798    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d35ced6_b776_4ff3_981c_fc3679d0fca2.slice/crio-0aa18bdfd7a7649bd7cfa7b2ca369544f7f2661cbdae8ce5dd33c822480753e6 WatchSource:0}: Error finding container 0aa18bdfd7a7649bd7cfa7b2ca369544f7f2661cbdae8ce5dd33c822480753e6: Status 404 returned error can't find the container with id 0aa18bdfd7a7649bd7cfa7b2ca369544f7f2661cbdae8ce5dd33c822480753e6
Sep 16 18:51:34.659026 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:34.658989    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x" event={"ID":"1d35ced6-b776-4ff3-981c-fc3679d0fca2","Type":"ContainerStarted","Data":"0aa18bdfd7a7649bd7cfa7b2ca369544f7f2661cbdae8ce5dd33c822480753e6"}
Sep 16 18:51:35.638496 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:35.638458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:35.771199 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:35.771160    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:36.395158 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:36.395106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:36.499241 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:36.498927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:36.599394 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:36.599296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:36.871632 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:36.871599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:37.011627 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:37.011591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:37.746071 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:37.746034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:37.870555 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:37.870527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:38.013411 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:38.013337    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:38.778585 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:38.778549    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:51:38.778710 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:38.778619    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:51:39.237211 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.237179    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:39.346715 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.346675    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:39.468119 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.468068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:39.679319 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.679230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x" event={"ID":"1d35ced6-b776-4ff3-981c-fc3679d0fca2","Type":"ContainerStarted","Data":"c6ebd3d13ab32902d415c9d5f897788e5c83d2f1df1849b1d71299415a51f17a"}
Sep 16 18:51:39.679605 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.679388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:51:39.680834 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.680808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n" event={"ID":"5193dacd-56cd-42d7-8284-73da8826fbf4","Type":"ContainerStarted","Data":"2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad"}
Sep 16 18:51:39.681012 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.680997    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:39.682904 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.682882    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:39.705392 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.705351    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x" podStartSLOduration=2.391639459 podStartE2EDuration="6.705338792s" totalImagesPullingTime="4.313699333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:39.703331172 +0000 UTC m=+732.071387093" watchObservedRunningTime="2026-09-16 18:51:39.705338792 +0000 UTC m=+732.073394687"
Sep 16 18:51:39.731671 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.731629    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n" podStartSLOduration=4.217382122 podStartE2EDuration="13.731616495s" totalImagesPullingTime="9.514234373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:39.731286216 +0000 UTC m=+732.099342137" watchObservedRunningTime="2026-09-16 18:51:39.731616495 +0000 UTC m=+732.099672396"
Sep 16 18:51:39.943651 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:39.943573    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"]
Sep 16 18:51:40.645200 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:40.645164    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-p8mvw"
Sep 16 18:51:41.687809 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.687759    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n" podUID="5193dacd-56cd-42d7-8284-73da8826fbf4" containerName="discovery" containerID="cri-o://2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad" gracePeriod=30
Sep 16 18:51:41.929252 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.929225    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:41.993753 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.993724    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-dns-cert\") pod \"5193dacd-56cd-42d7-8284-73da8826fbf4\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") "
Sep 16 18:51:41.993753 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.993757    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvpmn\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-kube-api-access-wvpmn\") pod \"5193dacd-56cd-42d7-8284-73da8826fbf4\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") "
Sep 16 18:51:41.993964 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.993784    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-cacerts\") pod \"5193dacd-56cd-42d7-8284-73da8826fbf4\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") "
Sep 16 18:51:41.993964 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.993801    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-ca-configmap\") pod \"5193dacd-56cd-42d7-8284-73da8826fbf4\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") "
Sep 16 18:51:41.993964 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.993844    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-kubeconfig\") pod \"5193dacd-56cd-42d7-8284-73da8826fbf4\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") "
Sep 16 18:51:41.993964 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.993867    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-token\") pod \"5193dacd-56cd-42d7-8284-73da8826fbf4\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") "
Sep 16 18:51:41.993964 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.993895    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5193dacd-56cd-42d7-8284-73da8826fbf4-local-certs\") pod \"5193dacd-56cd-42d7-8284-73da8826fbf4\" (UID: \"5193dacd-56cd-42d7-8284-73da8826fbf4\") "
Sep 16 18:51:41.994328 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.994299    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5193dacd-56cd-42d7-8284-73da8826fbf4" (UID: "5193dacd-56cd-42d7-8284-73da8826fbf4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:41.996178 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.996150    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-cacerts" (OuterVolumeSpecName: "cacerts") pod "5193dacd-56cd-42d7-8284-73da8826fbf4" (UID: "5193dacd-56cd-42d7-8284-73da8826fbf4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:41.996324 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.996303    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5193dacd-56cd-42d7-8284-73da8826fbf4-local-certs" (OuterVolumeSpecName: "local-certs") pod "5193dacd-56cd-42d7-8284-73da8826fbf4" (UID: "5193dacd-56cd-42d7-8284-73da8826fbf4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:41.996470 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.996440    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5193dacd-56cd-42d7-8284-73da8826fbf4" (UID: "5193dacd-56cd-42d7-8284-73da8826fbf4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:41.996564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.996508    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5193dacd-56cd-42d7-8284-73da8826fbf4" (UID: "5193dacd-56cd-42d7-8284-73da8826fbf4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:41.996564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.996534    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-kube-api-access-wvpmn" (OuterVolumeSpecName: "kube-api-access-wvpmn") pod "5193dacd-56cd-42d7-8284-73da8826fbf4" (UID: "5193dacd-56cd-42d7-8284-73da8826fbf4"). InnerVolumeSpecName "kube-api-access-wvpmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:41.996564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:41.996546    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5193dacd-56cd-42d7-8284-73da8826fbf4" (UID: "5193dacd-56cd-42d7-8284-73da8826fbf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:42.094874 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.094842    2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-kubeconfig\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:51:42.094874 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.094867    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-token\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:51:42.094874 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.094879    2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5193dacd-56cd-42d7-8284-73da8826fbf4-local-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:51:42.095139 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.094891    2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-dns-cert\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:51:42.095139 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.094902    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvpmn\" (UniqueName: \"kubernetes.io/projected/5193dacd-56cd-42d7-8284-73da8826fbf4-kube-api-access-wvpmn\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:51:42.095139 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.094914    2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5193dacd-56cd-42d7-8284-73da8826fbf4-cacerts\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:51:42.095139 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.094927    2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5193dacd-56cd-42d7-8284-73da8826fbf4-istio-csr-ca-configmap\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:51:42.655605 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.655570    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7kv5l"
Sep 16 18:51:42.692425 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.692392    2559 generic.go:356] "Generic (PLEG): container finished" podID="5193dacd-56cd-42d7-8284-73da8826fbf4" containerID="2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad" exitCode=0
Sep 16 18:51:42.692822 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.692435    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"
Sep 16 18:51:42.692822 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.692473    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n" event={"ID":"5193dacd-56cd-42d7-8284-73da8826fbf4","Type":"ContainerDied","Data":"2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad"}
Sep 16 18:51:42.692822 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.692518    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n" event={"ID":"5193dacd-56cd-42d7-8284-73da8826fbf4","Type":"ContainerDied","Data":"fa64412ed169e1874c6accdd3378f09fc3fc8b8da309a8a9d067bd27069c4a1d"}
Sep 16 18:51:42.692822 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.692540    2559 scope.go:117] "RemoveContainer" containerID="2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad"
Sep 16 18:51:42.700315 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.700297    2559 scope.go:117] "RemoveContainer" containerID="2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad"
Sep 16 18:51:42.700589 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:51:42.700571    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad\": container with ID starting with 2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad not found: ID does not exist" containerID="2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad"
Sep 16 18:51:42.700635 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.700597    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad"} err="failed to get container status \"2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad\": rpc error: code = NotFound desc = could not find container \"2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad\": container with ID starting with 2a2de3bceec35e2eccfd8d50029f584afb2451c470d34ac204bc483bb74203ad not found: ID does not exist"
Sep 16 18:51:42.897328 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.897298    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"]
Sep 16 18:51:42.922682 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:42.922618    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-czq7n"]
Sep 16 18:51:44.221301 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:44.221273    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5193dacd-56cd-42d7-8284-73da8826fbf4" path="/var/lib/kubelet/pods/5193dacd-56cd-42d7-8284-73da8826fbf4/volumes"
Sep 16 18:51:50.687378 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:51:50.687343    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f86d45999-zx52x"
Sep 16 18:52:25.371909 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:25.371874    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:25.474533 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:25.474504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:43.206453 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:43.206424    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:45.958579 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:45.958549    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:46.777582 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:46.777551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:47.596658 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:47.596625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:50.580506 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:50.580438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:51.004707 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:51.004676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:51.345110 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:51.345036    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:51.441927 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:51.441894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:53.592583 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:53.592554    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:53.831215 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:53.831185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:53.958983 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:53.958931    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:54.232461 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:54.232380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:54.333544 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:54.333513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:54.439082 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:54.439045    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:55.583169 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:55.583139    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:55.888038 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:55.887951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:57.271869 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.271839    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:57.646496 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.646420    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"]
Sep 16 18:52:57.646670 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.646657    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5193dacd-56cd-42d7-8284-73da8826fbf4" containerName="discovery"
Sep 16 18:52:57.646715 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.646671    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5193dacd-56cd-42d7-8284-73da8826fbf4" containerName="discovery"
Sep 16 18:52:57.646749 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.646724    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5193dacd-56cd-42d7-8284-73da8826fbf4" containerName="discovery"
Sep 16 18:52:57.648615 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.648592    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.649117 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.649094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:57.651503 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.651484    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 18:52:57.652097 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.652080    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rc6fs\""
Sep 16 18:52:57.652661 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.652644    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:52:57.652730 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.652665    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:52:57.652730 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.652651    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 18:52:57.660603 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.660580    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"]
Sep 16 18:52:57.742081 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.742042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mvjt\" (UniqueName: \"kubernetes.io/projected/d3417344-9071-486d-a528-de677568078e-kube-api-access-6mvjt\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.742292 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.742094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3417344-9071-486d-a528-de677568078e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.742292 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.742165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3417344-9071-486d-a528-de677568078e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.842960 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.842922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mvjt\" (UniqueName: \"kubernetes.io/projected/d3417344-9071-486d-a528-de677568078e-kube-api-access-6mvjt\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.843162 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.842988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3417344-9071-486d-a528-de677568078e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.843162 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.843013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3417344-9071-486d-a528-de677568078e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.843600 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.843580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d3417344-9071-486d-a528-de677568078e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.845404 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.845381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3417344-9071-486d-a528-de677568078e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.853197 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.853169    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mvjt\" (UniqueName: \"kubernetes.io/projected/d3417344-9071-486d-a528-de677568078e-kube-api-access-6mvjt\") pod \"kuadrant-console-plugin-fbdd4f445-hqpnf\" (UID: \"d3417344-9071-486d-a528-de677568078e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:57.896660 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.896578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:57.958177 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:57.958116    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"
Sep 16 18:52:58.251594 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:58.251560    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf"]
Sep 16 18:52:58.254345 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:52:58.254320    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3417344_9071_486d_a528_de677568078e.slice/crio-e35761fbd905af8bde205a3495959b465b77c7c2a26016ff87f608c9e7625c85 WatchSource:0}: Error finding container e35761fbd905af8bde205a3495959b465b77c7c2a26016ff87f608c9e7625c85: Status 404 returned error can't find the container with id e35761fbd905af8bde205a3495959b465b77c7c2a26016ff87f608c9e7625c85
Sep 16 18:52:58.973277 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:52:58.973241    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf" event={"ID":"d3417344-9071-486d-a528-de677568078e","Type":"ContainerStarted","Data":"e35761fbd905af8bde205a3495959b465b77c7c2a26016ff87f608c9e7625c85"}
Sep 16 18:53:01.032435 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:01.032400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:01.122195 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:01.122161    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:01.435925 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:01.435800    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:01.535925 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:01.535888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:01.788389 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:01.788359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:01.890153 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:01.890122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:04.997158 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:04.997117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf" event={"ID":"d3417344-9071-486d-a528-de677568078e","Type":"ContainerStarted","Data":"ce779112a5e8838e4a7cde04b8115bdb065a676f8a4fa536db18e27050e17324"}
Sep 16 18:53:05.019443 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:05.019390    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hqpnf" podStartSLOduration=2.063262489 podStartE2EDuration="8.01937742s" totalImagesPullingTime="5.956114931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:05.017850941 +0000 UTC m=+817.385906842" watchObservedRunningTime="2026-09-16 18:53:05.01937742 +0000 UTC m=+817.387433320"
Sep 16 18:53:06.157590 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:06.157556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:09.797539 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:09.797508    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:09.897108 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:09.897070    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:12.249473 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:12.249438    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:12.320248 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:12.320211    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:17.047908 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:17.047871    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:17.177333 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:17.177300    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:22.205991 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:22.205941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:22.312819 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:22.312781    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:22.669438 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:22.669360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:22.753948 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:22.753911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:22.964959 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:22.964930    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:23.054490 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:23.054462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:27.725445 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:27.725409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:27.814244 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:27.814213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:29.544430 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:29.544387    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:31.252384 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:31.252355    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:31.358137 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:31.358104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:31.459236 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:31.459207    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:33.726131 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:33.726098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:33.829718 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:33.829689    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:38.413142 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:38.413109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:38.540987 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:38.540951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:38.853219 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:38.853188    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:38.962451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:38.962420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:43.866436 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:43.866401    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:44.093948 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:44.093918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:45.649322 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:45.649292    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:45.757866 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:45.757832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:45.815895 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:45.815863    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:46.534052 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.534021    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dcqrr"]
Sep 16 18:53:46.551004 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.550955    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dcqrr"]
Sep 16 18:53:46.551130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.551091    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr"
Sep 16 18:53:46.553838 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.553819    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ms2sb\""
Sep 16 18:53:46.601704 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.601670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:46.617665 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.617640    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km2f5\" (UniqueName: \"kubernetes.io/projected/73427b8b-522d-414b-91f0-f6f4105201a3-kube-api-access-km2f5\") pod \"authorino-7bcfd8f9cd-dcqrr\" (UID: \"73427b8b-522d-414b-91f0-f6f4105201a3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr"
Sep 16 18:53:46.660625 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.660586    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-6hzmj"]
Sep 16 18:53:46.663615 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.663592    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6hzmj"
Sep 16 18:53:46.679946 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.679919    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6hzmj"]
Sep 16 18:53:46.700953 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.700921    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:46.718026 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.718000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km2f5\" (UniqueName: \"kubernetes.io/projected/73427b8b-522d-414b-91f0-f6f4105201a3-kube-api-access-km2f5\") pod \"authorino-7bcfd8f9cd-dcqrr\" (UID: \"73427b8b-522d-414b-91f0-f6f4105201a3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr"
Sep 16 18:53:46.718114 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.718063    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tts\" (UniqueName: \"kubernetes.io/projected/c4ab5f42-15ef-4697-a639-93077e455d3a-kube-api-access-v2tts\") pod \"authorino-59c974bd48-6hzmj\" (UID: \"c4ab5f42-15ef-4697-a639-93077e455d3a\") " pod="kuadrant-system/authorino-59c974bd48-6hzmj"
Sep 16 18:53:46.730546 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.730511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km2f5\" (UniqueName: \"kubernetes.io/projected/73427b8b-522d-414b-91f0-f6f4105201a3-kube-api-access-km2f5\") pod \"authorino-7bcfd8f9cd-dcqrr\" (UID: \"73427b8b-522d-414b-91f0-f6f4105201a3\") " pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr"
Sep 16 18:53:46.819410 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.819325    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tts\" (UniqueName: \"kubernetes.io/projected/c4ab5f42-15ef-4697-a639-93077e455d3a-kube-api-access-v2tts\") pod \"authorino-59c974bd48-6hzmj\" (UID: \"c4ab5f42-15ef-4697-a639-93077e455d3a\") " pod="kuadrant-system/authorino-59c974bd48-6hzmj"
Sep 16 18:53:46.828862 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.828829    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tts\" (UniqueName: \"kubernetes.io/projected/c4ab5f42-15ef-4697-a639-93077e455d3a-kube-api-access-v2tts\") pod \"authorino-59c974bd48-6hzmj\" (UID: \"c4ab5f42-15ef-4697-a639-93077e455d3a\") " pod="kuadrant-system/authorino-59c974bd48-6hzmj"
Sep 16 18:53:46.860636 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.860597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr"
Sep 16 18:53:46.971762 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:46.971727    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6hzmj"
Sep 16 18:53:47.127431 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:47.127408    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dcqrr"]
Sep 16 18:53:47.129875 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:53:47.129847    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73427b8b_522d_414b_91f0_f6f4105201a3.slice/crio-6e440ee77ce6ebeff90735ccabe606bb4ca33af7da5d2b343a8d35a65485a523 WatchSource:0}: Error finding container 6e440ee77ce6ebeff90735ccabe606bb4ca33af7da5d2b343a8d35a65485a523: Status 404 returned error can't find the container with id 6e440ee77ce6ebeff90735ccabe606bb4ca33af7da5d2b343a8d35a65485a523
Sep 16 18:53:47.149038 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:47.149002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr" event={"ID":"73427b8b-522d-414b-91f0-f6f4105201a3","Type":"ContainerStarted","Data":"6e440ee77ce6ebeff90735ccabe606bb4ca33af7da5d2b343a8d35a65485a523"}
Sep 16 18:53:47.243984 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:47.239743    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6hzmj"]
Sep 16 18:53:48.155994 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:48.155942    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6hzmj" event={"ID":"c4ab5f42-15ef-4697-a639-93077e455d3a","Type":"ContainerStarted","Data":"666cb2b2c1bde1955f2d8b3a94a04293a84f43685ea128130407a6322edec622"}
Sep 16 18:53:50.165551 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:50.165516    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6hzmj" event={"ID":"c4ab5f42-15ef-4697-a639-93077e455d3a","Type":"ContainerStarted","Data":"571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd"}
Sep 16 18:53:50.167028 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:50.167002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr" event={"ID":"73427b8b-522d-414b-91f0-f6f4105201a3","Type":"ContainerStarted","Data":"3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9"}
Sep 16 18:53:50.185862 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:50.185818    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-6hzmj" podStartSLOduration=1.867918431 podStartE2EDuration="4.185804973s" totalImagesPullingTime="2.317886542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:50.18551882 +0000 UTC m=+862.553574721" watchObservedRunningTime="2026-09-16 18:53:50.185804973 +0000 UTC m=+862.553860874"
Sep 16 18:53:50.204485 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:50.204440    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr" podStartSLOduration=1.782975214 podStartE2EDuration="4.204426506s" totalImagesPullingTime="2.421451292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:50.202872994 +0000 UTC m=+862.570928899" watchObservedRunningTime="2026-09-16 18:53:50.204426506 +0000 UTC m=+862.572482407"
Sep 16 18:53:50.238498 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:50.238468    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dcqrr"]
Sep 16 18:53:52.177886 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:52.175820    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr" podUID="73427b8b-522d-414b-91f0-f6f4105201a3" containerName="authorino" containerID="cri-o://3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9" gracePeriod=30
Sep 16 18:53:52.421582 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:52.421556    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr"
Sep 16 18:53:52.566064 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:52.566027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km2f5\" (UniqueName: \"kubernetes.io/projected/73427b8b-522d-414b-91f0-f6f4105201a3-kube-api-access-km2f5\") pod \"73427b8b-522d-414b-91f0-f6f4105201a3\" (UID: \"73427b8b-522d-414b-91f0-f6f4105201a3\") "
Sep 16 18:53:52.568172 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:52.568144    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73427b8b-522d-414b-91f0-f6f4105201a3-kube-api-access-km2f5" (OuterVolumeSpecName: "kube-api-access-km2f5") pod "73427b8b-522d-414b-91f0-f6f4105201a3" (UID: "73427b8b-522d-414b-91f0-f6f4105201a3"). InnerVolumeSpecName "kube-api-access-km2f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:52.667461 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:52.667430    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km2f5\" (UniqueName: \"kubernetes.io/projected/73427b8b-522d-414b-91f0-f6f4105201a3-kube-api-access-km2f5\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:53:53.180484 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.180445    2559 generic.go:356] "Generic (PLEG): container finished" podID="73427b8b-522d-414b-91f0-f6f4105201a3" containerID="3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9" exitCode=0
Sep 16 18:53:53.180484 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.180485    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr"
Sep 16 18:53:53.180930 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.180525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr" event={"ID":"73427b8b-522d-414b-91f0-f6f4105201a3","Type":"ContainerDied","Data":"3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9"}
Sep 16 18:53:53.180930 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.180570    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-dcqrr" event={"ID":"73427b8b-522d-414b-91f0-f6f4105201a3","Type":"ContainerDied","Data":"6e440ee77ce6ebeff90735ccabe606bb4ca33af7da5d2b343a8d35a65485a523"}
Sep 16 18:53:53.180930 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.180587    2559 scope.go:117] "RemoveContainer" containerID="3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9"
Sep 16 18:53:53.188928 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.188910    2559 scope.go:117] "RemoveContainer" containerID="3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9"
Sep 16 18:53:53.189198 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:53:53.189176    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9\": container with ID starting with 3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9 not found: ID does not exist" containerID="3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9"
Sep 16 18:53:53.189266 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.189205    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9"} err="failed to get container status \"3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9\": rpc error: code = NotFound desc = could not find container \"3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9\": container with ID starting with 3598d3d70c8135305fbf8286080c7b8455d668b50dacb2f2c355cf3fcd24dcf9 not found: ID does not exist"
Sep 16 18:53:53.201485 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.201464    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dcqrr"]
Sep 16 18:53:53.205305 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:53.205283    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-dcqrr"]
Sep 16 18:53:54.221339 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:53:54.221299    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73427b8b-522d-414b-91f0-f6f4105201a3" path="/var/lib/kubelet/pods/73427b8b-522d-414b-91f0-f6f4105201a3/volumes"
Sep 16 18:54:01.777104 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:01.777061    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:01.869128 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:01.869098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:02.334550 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:02.334520    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:15.080380 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.080349    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:15.232471 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.232435    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:15.379553 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.379475    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-68qzw"]
Sep 16 18:54:15.379872 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.379856    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73427b8b-522d-414b-91f0-f6f4105201a3" containerName="authorino"
Sep 16 18:54:15.379925 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.379876    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73427b8b-522d-414b-91f0-f6f4105201a3" containerName="authorino"
Sep 16 18:54:15.379965 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.379954    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="73427b8b-522d-414b-91f0-f6f4105201a3" containerName="authorino"
Sep 16 18:54:15.382901 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.382876    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-68qzw"
Sep 16 18:54:15.400787 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.400763    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-68qzw"]
Sep 16 18:54:15.449774 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.449736    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-68qzw"]
Sep 16 18:54:15.450017 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:54:15.449993    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wcfg8], unattached volumes=[], failed to process volumes=[kube-api-access-wcfg8]: context canceled" pod="kuadrant-system/authorino-557f876f69-68qzw" podUID="f378c90a-84ea-42cb-875d-f3354a92c305"
Sep 16 18:54:15.479836 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.479808    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-brwkc"]
Sep 16 18:54:15.482561 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.482546    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:15.485249 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.485232    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 18:54:15.494018 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.493996    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-brwkc"]
Sep 16 18:54:15.540967 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.540939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcfg8\" (UniqueName: \"kubernetes.io/projected/f378c90a-84ea-42cb-875d-f3354a92c305-kube-api-access-wcfg8\") pod \"authorino-557f876f69-68qzw\" (UID: \"f378c90a-84ea-42cb-875d-f3354a92c305\") " pod="kuadrant-system/authorino-557f876f69-68qzw"
Sep 16 18:54:15.642329 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.642228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlkb\" (UniqueName: \"kubernetes.io/projected/0975c661-43c9-4df0-bd52-41e04387f77a-kube-api-access-hnlkb\") pod \"authorino-6d4dfc649f-brwkc\" (UID: \"0975c661-43c9-4df0-bd52-41e04387f77a\") " pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:15.642329 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.642304    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0975c661-43c9-4df0-bd52-41e04387f77a-tls-cert\") pod \"authorino-6d4dfc649f-brwkc\" (UID: \"0975c661-43c9-4df0-bd52-41e04387f77a\") " pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:15.642505 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.642341    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcfg8\" (UniqueName: \"kubernetes.io/projected/f378c90a-84ea-42cb-875d-f3354a92c305-kube-api-access-wcfg8\") pod \"authorino-557f876f69-68qzw\" (UID: \"f378c90a-84ea-42cb-875d-f3354a92c305\") " pod="kuadrant-system/authorino-557f876f69-68qzw"
Sep 16 18:54:15.655645 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.655614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcfg8\" (UniqueName: \"kubernetes.io/projected/f378c90a-84ea-42cb-875d-f3354a92c305-kube-api-access-wcfg8\") pod \"authorino-557f876f69-68qzw\" (UID: \"f378c90a-84ea-42cb-875d-f3354a92c305\") " pod="kuadrant-system/authorino-557f876f69-68qzw"
Sep 16 18:54:15.660854 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.660827    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-brwkc"]
Sep 16 18:54:15.661100 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:54:15.661071    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hnlkb tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-brwkc" podUID="0975c661-43c9-4df0-bd52-41e04387f77a"
Sep 16 18:54:15.699612 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.699582    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85bdf675cb-2lfww"]
Sep 16 18:54:15.702491 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.702475    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:54:15.717379 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.717359    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85bdf675cb-2lfww"]
Sep 16 18:54:15.743695 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.743671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0975c661-43c9-4df0-bd52-41e04387f77a-tls-cert\") pod \"authorino-6d4dfc649f-brwkc\" (UID: \"0975c661-43c9-4df0-bd52-41e04387f77a\") " pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:15.743802 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.743718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlkb\" (UniqueName: \"kubernetes.io/projected/0975c661-43c9-4df0-bd52-41e04387f77a-kube-api-access-hnlkb\") pod \"authorino-6d4dfc649f-brwkc\" (UID: \"0975c661-43c9-4df0-bd52-41e04387f77a\") " pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:15.745945 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.745928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0975c661-43c9-4df0-bd52-41e04387f77a-tls-cert\") pod \"authorino-6d4dfc649f-brwkc\" (UID: \"0975c661-43c9-4df0-bd52-41e04387f77a\") " pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:15.753432 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.753409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlkb\" (UniqueName: \"kubernetes.io/projected/0975c661-43c9-4df0-bd52-41e04387f77a-kube-api-access-hnlkb\") pod \"authorino-6d4dfc649f-brwkc\" (UID: \"0975c661-43c9-4df0-bd52-41e04387f77a\") " pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:15.844943 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.844910    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/051c485a-1e7b-4476-b7f7-7f29852d4124-tls-cert\") pod \"authorino-85bdf675cb-2lfww\" (UID: \"051c485a-1e7b-4476-b7f7-7f29852d4124\") " pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:54:15.844943 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.844944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcngx\" (UniqueName: \"kubernetes.io/projected/051c485a-1e7b-4476-b7f7-7f29852d4124-kube-api-access-tcngx\") pod \"authorino-85bdf675cb-2lfww\" (UID: \"051c485a-1e7b-4476-b7f7-7f29852d4124\") " pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:54:15.946043 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.945950    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/051c485a-1e7b-4476-b7f7-7f29852d4124-tls-cert\") pod \"authorino-85bdf675cb-2lfww\" (UID: \"051c485a-1e7b-4476-b7f7-7f29852d4124\") " pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:54:15.946043 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.946018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcngx\" (UniqueName: \"kubernetes.io/projected/051c485a-1e7b-4476-b7f7-7f29852d4124-kube-api-access-tcngx\") pod \"authorino-85bdf675cb-2lfww\" (UID: \"051c485a-1e7b-4476-b7f7-7f29852d4124\") " pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:54:15.948369 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.948341    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/051c485a-1e7b-4476-b7f7-7f29852d4124-tls-cert\") pod \"authorino-85bdf675cb-2lfww\" (UID: \"051c485a-1e7b-4476-b7f7-7f29852d4124\") " pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:54:15.955144 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:15.955121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcngx\" (UniqueName: \"kubernetes.io/projected/051c485a-1e7b-4476-b7f7-7f29852d4124-kube-api-access-tcngx\") pod \"authorino-85bdf675cb-2lfww\" (UID: \"051c485a-1e7b-4476-b7f7-7f29852d4124\") " pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:54:16.011101 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.011070    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:54:16.262782 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.262741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:16.263206 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.262741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-68qzw"
Sep 16 18:54:16.270478 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.270456    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:16.272516 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.272503    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-68qzw"
Sep 16 18:54:16.282857 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.282835    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85bdf675cb-2lfww"]
Sep 16 18:54:16.285683 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:54:16.285661    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod051c485a_1e7b_4476_b7f7_7f29852d4124.slice/crio-5632e22e330e00781e7fa6168941550d1fb9cf211ca948fe4efce594193db552 WatchSource:0}: Error finding container 5632e22e330e00781e7fa6168941550d1fb9cf211ca948fe4efce594193db552: Status 404 returned error can't find the container with id 5632e22e330e00781e7fa6168941550d1fb9cf211ca948fe4efce594193db552
Sep 16 18:54:16.449802 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.449773    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnlkb\" (UniqueName: \"kubernetes.io/projected/0975c661-43c9-4df0-bd52-41e04387f77a-kube-api-access-hnlkb\") pod \"0975c661-43c9-4df0-bd52-41e04387f77a\" (UID: \"0975c661-43c9-4df0-bd52-41e04387f77a\") "
Sep 16 18:54:16.449959 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.449816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0975c661-43c9-4df0-bd52-41e04387f77a-tls-cert\") pod \"0975c661-43c9-4df0-bd52-41e04387f77a\" (UID: \"0975c661-43c9-4df0-bd52-41e04387f77a\") "
Sep 16 18:54:16.449959 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.449845    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcfg8\" (UniqueName: \"kubernetes.io/projected/f378c90a-84ea-42cb-875d-f3354a92c305-kube-api-access-wcfg8\") pod \"f378c90a-84ea-42cb-875d-f3354a92c305\" (UID: \"f378c90a-84ea-42cb-875d-f3354a92c305\") "
Sep 16 18:54:16.452062 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.452022    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0975c661-43c9-4df0-bd52-41e04387f77a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0975c661-43c9-4df0-bd52-41e04387f77a" (UID: "0975c661-43c9-4df0-bd52-41e04387f77a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:54:16.452180 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.452082    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f378c90a-84ea-42cb-875d-f3354a92c305-kube-api-access-wcfg8" (OuterVolumeSpecName: "kube-api-access-wcfg8") pod "f378c90a-84ea-42cb-875d-f3354a92c305" (UID: "f378c90a-84ea-42cb-875d-f3354a92c305"). InnerVolumeSpecName "kube-api-access-wcfg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:16.452180 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.452119    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0975c661-43c9-4df0-bd52-41e04387f77a-kube-api-access-hnlkb" (OuterVolumeSpecName: "kube-api-access-hnlkb") pod "0975c661-43c9-4df0-bd52-41e04387f77a" (UID: "0975c661-43c9-4df0-bd52-41e04387f77a"). InnerVolumeSpecName "kube-api-access-hnlkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:16.551295 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.551214    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnlkb\" (UniqueName: \"kubernetes.io/projected/0975c661-43c9-4df0-bd52-41e04387f77a-kube-api-access-hnlkb\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:54:16.551295 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.551242    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0975c661-43c9-4df0-bd52-41e04387f77a-tls-cert\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:54:16.551295 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:16.551252    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcfg8\" (UniqueName: \"kubernetes.io/projected/f378c90a-84ea-42cb-875d-f3354a92c305-kube-api-access-wcfg8\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:54:17.268510 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:17.268473    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85bdf675cb-2lfww" event={"ID":"051c485a-1e7b-4476-b7f7-7f29852d4124","Type":"ContainerStarted","Data":"5632e22e330e00781e7fa6168941550d1fb9cf211ca948fe4efce594193db552"}
Sep 16 18:54:17.268510 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:17.268499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-68qzw"
Sep 16 18:54:17.268510 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:17.268482    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-brwkc"
Sep 16 18:54:17.302576 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:17.302549    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-68qzw"]
Sep 16 18:54:17.308748 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:17.308723    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-68qzw"]
Sep 16 18:54:17.339089 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:17.339062    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-brwkc"]
Sep 16 18:54:17.346471 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:17.346449    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-brwkc"]
Sep 16 18:54:18.221930 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.221896    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0975c661-43c9-4df0-bd52-41e04387f77a" path="/var/lib/kubelet/pods/0975c661-43c9-4df0-bd52-41e04387f77a/volumes"
Sep 16 18:54:18.222242 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.222226    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f378c90a-84ea-42cb-875d-f3354a92c305" path="/var/lib/kubelet/pods/f378c90a-84ea-42cb-875d-f3354a92c305/volumes"
Sep 16 18:54:18.273525 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.273488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85bdf675cb-2lfww" event={"ID":"051c485a-1e7b-4476-b7f7-7f29852d4124","Type":"ContainerStarted","Data":"d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3"}
Sep 16 18:54:18.291953 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.291898    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85bdf675cb-2lfww" podStartSLOduration=2.224991402 podStartE2EDuration="3.29188342s" totalImagesPullingTime="1.066892018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:18.289914803 +0000 UTC m=+890.657970704" watchObservedRunningTime="2026-09-16 18:54:18.29188342 +0000 UTC m=+890.659939321"
Sep 16 18:54:18.322554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.322522    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6hzmj"]
Sep 16 18:54:18.322703 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.322683    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-6hzmj" podUID="c4ab5f42-15ef-4697-a639-93077e455d3a" containerName="authorino" containerID="cri-o://571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd" gracePeriod=30
Sep 16 18:54:18.571362 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.571339    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6hzmj"
Sep 16 18:54:18.667771 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.667724    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2tts\" (UniqueName: \"kubernetes.io/projected/c4ab5f42-15ef-4697-a639-93077e455d3a-kube-api-access-v2tts\") pod \"c4ab5f42-15ef-4697-a639-93077e455d3a\" (UID: \"c4ab5f42-15ef-4697-a639-93077e455d3a\") "
Sep 16 18:54:18.669808 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.669775    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ab5f42-15ef-4697-a639-93077e455d3a-kube-api-access-v2tts" (OuterVolumeSpecName: "kube-api-access-v2tts") pod "c4ab5f42-15ef-4697-a639-93077e455d3a" (UID: "c4ab5f42-15ef-4697-a639-93077e455d3a"). InnerVolumeSpecName "kube-api-access-v2tts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:18.773166 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:18.769335    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tts\" (UniqueName: \"kubernetes.io/projected/c4ab5f42-15ef-4697-a639-93077e455d3a-kube-api-access-v2tts\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:54:19.278053 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.278021    2559 generic.go:356] "Generic (PLEG): container finished" podID="c4ab5f42-15ef-4697-a639-93077e455d3a" containerID="571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd" exitCode=0
Sep 16 18:54:19.278509 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.278090    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-6hzmj"
Sep 16 18:54:19.278509 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.278113    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6hzmj" event={"ID":"c4ab5f42-15ef-4697-a639-93077e455d3a","Type":"ContainerDied","Data":"571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd"}
Sep 16 18:54:19.278509 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.278154    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-6hzmj" event={"ID":"c4ab5f42-15ef-4697-a639-93077e455d3a","Type":"ContainerDied","Data":"666cb2b2c1bde1955f2d8b3a94a04293a84f43685ea128130407a6322edec622"}
Sep 16 18:54:19.278509 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.278171    2559 scope.go:117] "RemoveContainer" containerID="571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd"
Sep 16 18:54:19.286854 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.286839    2559 scope.go:117] "RemoveContainer" containerID="571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd"
Sep 16 18:54:19.287159 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:54:19.287138    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd\": container with ID starting with 571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd not found: ID does not exist" containerID="571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd"
Sep 16 18:54:19.287232 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.287166    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd"} err="failed to get container status \"571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd\": rpc error: code = NotFound desc = could not find container \"571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd\": container with ID starting with 571ef7db59495b7b500b78b9e49c3e40f03ce4ea3564b1522f725d701b7b2bfd not found: ID does not exist"
Sep 16 18:54:19.302489 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.302467    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6hzmj"]
Sep 16 18:54:19.307590 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:19.307570    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-6hzmj"]
Sep 16 18:54:20.221544 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:20.221447    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4ab5f42-15ef-4697-a639-93077e455d3a" path="/var/lib/kubelet/pods/c4ab5f42-15ef-4697-a639-93077e455d3a/volumes"
Sep 16 18:54:23.368214 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:23.368185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:23.452190 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:23.452152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:23.528188 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:23.528158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:23.613140 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:23.613112    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:23.743861 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:23.743828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:27.134066 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:27.134028    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:27.235960 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:27.235923    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:28.010253 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:28.010229    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:54:28.010951 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:28.010931    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:54:58.732074 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:58.732039    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:58.822572 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:54:58.822541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:00.316143 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.316105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:00.415444 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.415413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:00.474498 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.474464    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:00.852610 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.852574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:00.952998 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.952939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:00.982632 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.982597    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w66f9"]
Sep 16 18:55:00.982877 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.982864    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ab5f42-15ef-4697-a639-93077e455d3a" containerName="authorino"
Sep 16 18:55:00.982928 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.982878    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ab5f42-15ef-4697-a639-93077e455d3a" containerName="authorino"
Sep 16 18:55:00.982991 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.982929    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ab5f42-15ef-4697-a639-93077e455d3a" containerName="authorino"
Sep 16 18:55:00.985535 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.985508    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:00.988361 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.988202    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5kw74\""
Sep 16 18:55:00.988509 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:00.988487    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 18:55:01.007262 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.007231    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w66f9"]
Sep 16 18:55:01.088936 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.088903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnlb\" (UniqueName: \"kubernetes.io/projected/3f1972d0-318d-4e9a-b367-6745577a2e9a-kube-api-access-7bnlb\") pod \"payload-processing-5b54b7fbb7-w66f9\" (UID: \"3f1972d0-318d-4e9a-b367-6745577a2e9a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:01.089139 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.088968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1972d0-318d-4e9a-b367-6745577a2e9a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w66f9\" (UID: \"3f1972d0-318d-4e9a-b367-6745577a2e9a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:01.190157 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.190059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1972d0-318d-4e9a-b367-6745577a2e9a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w66f9\" (UID: \"3f1972d0-318d-4e9a-b367-6745577a2e9a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:01.190157 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.190135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnlb\" (UniqueName: \"kubernetes.io/projected/3f1972d0-318d-4e9a-b367-6745577a2e9a-kube-api-access-7bnlb\") pod \"payload-processing-5b54b7fbb7-w66f9\" (UID: \"3f1972d0-318d-4e9a-b367-6745577a2e9a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:01.190932 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.190900    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1972d0-318d-4e9a-b367-6745577a2e9a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-w66f9\" (UID: \"3f1972d0-318d-4e9a-b367-6745577a2e9a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:01.200074 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.200047    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnlb\" (UniqueName: \"kubernetes.io/projected/3f1972d0-318d-4e9a-b367-6745577a2e9a-kube-api-access-7bnlb\") pod \"payload-processing-5b54b7fbb7-w66f9\" (UID: \"3f1972d0-318d-4e9a-b367-6745577a2e9a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:01.296910 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.296868    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:01.566091 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:01.566066    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-w66f9"]
Sep 16 18:55:01.568524 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:55:01.568499    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1972d0_318d_4e9a_b367_6745577a2e9a.slice/crio-d800d7fb5e033bbc33565174547fc6164773588bd56903481a09041db100faef WatchSource:0}: Error finding container d800d7fb5e033bbc33565174547fc6164773588bd56903481a09041db100faef: Status 404 returned error can't find the container with id d800d7fb5e033bbc33565174547fc6164773588bd56903481a09041db100faef
Sep 16 18:55:02.449987 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:02.449945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9" event={"ID":"3f1972d0-318d-4e9a-b367-6745577a2e9a","Type":"ContainerStarted","Data":"d800d7fb5e033bbc33565174547fc6164773588bd56903481a09041db100faef"}
Sep 16 18:55:05.463185 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:05.463143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9" event={"ID":"3f1972d0-318d-4e9a-b367-6745577a2e9a","Type":"ContainerStarted","Data":"e64c41b65680a7aba1014aa30afb1db73ce536aad35eb7e4a2207cba16a431b3"}
Sep 16 18:55:05.463585 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:05.463220    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:05.486153 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:05.486092    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9" podStartSLOduration=2.250987565 podStartE2EDuration="5.486073932s" totalImagesPullingTime="3.235086367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:05.482835188 +0000 UTC m=+937.850891098" watchObservedRunningTime="2026-09-16 18:55:05.486073932 +0000 UTC m=+937.854129837"
Sep 16 18:55:10.618153 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:10.618105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:10.716040 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:10.716002    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:16.468455 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:16.468425    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-w66f9"
Sep 16 18:55:18.908847 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:18.908818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:33.523592 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:33.523556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:40.611543 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:40.611511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:57.473855 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:55:57.473774    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:04.302021 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.301989    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"]
Sep 16 18:56:04.305207 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.305191    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.309612 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.309582    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lhtq7\""
Sep 16 18:56:04.309612 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.309602    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 18:56:04.309799 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.309671    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 18:56:04.309799 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.309770    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 16 18:56:04.318230 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.318206    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"]
Sep 16 18:56:04.360402 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.360370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:04.368838 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.368810    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf79e50e-a09a-4b24-8db6-900f209eb414-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.368961 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.368851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.368961 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.368911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.369056 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.368965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbddw\" (UniqueName: \"kubernetes.io/projected/bf79e50e-a09a-4b24-8db6-900f209eb414-kube-api-access-qbddw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.369056 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.369020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.369056 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.369042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.469895 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.469862    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbddw\" (UniqueName: \"kubernetes.io/projected/bf79e50e-a09a-4b24-8db6-900f209eb414-kube-api-access-qbddw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.470120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.469911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.470120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.469939    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.470120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.470009    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf79e50e-a09a-4b24-8db6-900f209eb414-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.470120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.470034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.470120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.470057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.470392 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.470369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.470461 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.470399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.470461 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.470430    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.472182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.472162    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf79e50e-a09a-4b24-8db6-900f209eb414-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.472361 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.472343    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf79e50e-a09a-4b24-8db6-900f209eb414-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.479286 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.479265    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbddw\" (UniqueName: \"kubernetes.io/projected/bf79e50e-a09a-4b24-8db6-900f209eb414-kube-api-access-qbddw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-4vpp2\" (UID: \"bf79e50e-a09a-4b24-8db6-900f209eb414\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.615160 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.615073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:04.889860 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.889771    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"]
Sep 16 18:56:04.893168 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:56:04.893140    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf79e50e_a09a_4b24_8db6_900f209eb414.slice/crio-c3986ef382804063ea5d0e9a685fc7741afdf8ca8834dce2f7d72f3cdbe588b0 WatchSource:0}: Error finding container c3986ef382804063ea5d0e9a685fc7741afdf8ca8834dce2f7d72f3cdbe588b0: Status 404 returned error can't find the container with id c3986ef382804063ea5d0e9a685fc7741afdf8ca8834dce2f7d72f3cdbe588b0
Sep 16 18:56:04.894460 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:04.894442    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:56:05.693671 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:05.693629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2" event={"ID":"bf79e50e-a09a-4b24-8db6-900f209eb414","Type":"ContainerStarted","Data":"c3986ef382804063ea5d0e9a685fc7741afdf8ca8834dce2f7d72f3cdbe588b0"}
Sep 16 18:56:06.698401 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:06.698367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2" event={"ID":"bf79e50e-a09a-4b24-8db6-900f209eb414","Type":"ContainerStarted","Data":"ed674234dd89baf9fe9b02c0cee0557139aef7b3e99031968b917400028947d5"}
Sep 16 18:56:06.735147 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:06.735079    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2" podStartSLOduration=1.707656147 podStartE2EDuration="2.735058185s" totalImagesPullingTime="1.027402038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:06.734695167 +0000 UTC m=+999.102751069" watchObservedRunningTime="2026-09-16 18:56:06.735058185 +0000 UTC m=+999.103114088"
Sep 16 18:56:09.479341 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.479307    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"]
Sep 16 18:56:09.482499 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.482478    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.485248 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.485224    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 18:56:09.496924 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.496897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"]
Sep 16 18:56:09.507305 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.507285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:09.612106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.612073    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.612106 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.612104    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.612286 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.612137    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.612286 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.612220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.612286 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.612260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.612385 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.612286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vt2\" (UniqueName: \"kubernetes.io/projected/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-kube-api-access-j7vt2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.712902 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.712868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vt2\" (UniqueName: \"kubernetes.io/projected/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-kube-api-access-j7vt2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.712902 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.712904    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.713135 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.712921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.713135 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.712956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.713135 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.713010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.713135 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.713036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.713354 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.713317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.713354 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.713347    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.713498 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.713477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.715223 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.715203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.715636 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.715616    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.723982 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.723951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vt2\" (UniqueName: \"kubernetes.io/projected/cb802bbf-8a7e-47fb-83e7-3416b556f4c2-kube-api-access-j7vt2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff\" (UID: \"cb802bbf-8a7e-47fb-83e7-3416b556f4c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:09.792296 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:09.792225    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:10.061543 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:10.061395    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"]
Sep 16 18:56:10.064095 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:56:10.064068    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb802bbf_8a7e_47fb_83e7_3416b556f4c2.slice/crio-88a41d9f11e086eb07fb7a8341cff9e3e8f0030c767b7ce16d6e17337d6818f6 WatchSource:0}: Error finding container 88a41d9f11e086eb07fb7a8341cff9e3e8f0030c767b7ce16d6e17337d6818f6: Status 404 returned error can't find the container with id 88a41d9f11e086eb07fb7a8341cff9e3e8f0030c767b7ce16d6e17337d6818f6
Sep 16 18:56:10.714502 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:10.714467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff" event={"ID":"cb802bbf-8a7e-47fb-83e7-3416b556f4c2","Type":"ContainerStarted","Data":"09cf305ceb3be37155209a5c8c8b8581b7395ab159a3a8aafc2997aff6785b55"}
Sep 16 18:56:10.714854 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:10.714524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff" event={"ID":"cb802bbf-8a7e-47fb-83e7-3416b556f4c2","Type":"ContainerStarted","Data":"88a41d9f11e086eb07fb7a8341cff9e3e8f0030c767b7ce16d6e17337d6818f6"}
Sep 16 18:56:10.736040 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:10.735991    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff" podStartSLOduration=1.545278892 podStartE2EDuration="1.735961656s" totalImagesPullingTime="190.682764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:10.735142195 +0000 UTC m=+1003.103198116" watchObservedRunningTime="2026-09-16 18:56:10.735961656 +0000 UTC m=+1003.104017589"
Sep 16 18:56:14.615823 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:14.615782    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:14.616244 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:14.615846    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:14.628030 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:14.628007    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:14.742156 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:14.742123    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-4vpp2"
Sep 16 18:56:15.674314 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:15.674282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:19.792375 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:19.792333    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:19.792751 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:19.792394    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:19.804955 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:19.804924    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:20.766529 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:20.766497    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff"
Sep 16 18:56:26.941923 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:26.941891    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:31.473838 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.473803    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"]
Sep 16 18:56:31.482698 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.482675    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.485401 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.485382    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 18:56:31.489620 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.489592    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"]
Sep 16 18:56:31.515084 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.515053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:31.587566 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.587530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.587684 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.587574    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.587684 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.587600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.587767 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.587699    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pb6q\" (UniqueName: \"kubernetes.io/projected/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-kube-api-access-6pb6q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.587767 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.587729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.587767 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.587762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.688914 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.688877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.689097 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.688930    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.689097 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.688957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.689097 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.689042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pb6q\" (UniqueName: \"kubernetes.io/projected/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-kube-api-access-6pb6q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.689097 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.689068    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.689317 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.689124    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.689375 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.689347    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.689375 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.689361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.689469 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.689416    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.691287 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.691264    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.691387 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.691320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.699571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.699551    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pb6q\" (UniqueName: \"kubernetes.io/projected/cb38dfd2-ec66-4fa9-b025-bb4cc22f967d-kube-api-access-6pb6q\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-czfjm\" (UID: \"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:31.791837 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:31.791765    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:32.067871 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:32.067843    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"]
Sep 16 18:56:32.070325 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:56:32.070300    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb38dfd2_ec66_4fa9_b025_bb4cc22f967d.slice/crio-f33c774e3734db4bc8e32c4f1c7baa65f3a1ad81acb1d2f13089356316b4aa9c WatchSource:0}: Error finding container f33c774e3734db4bc8e32c4f1c7baa65f3a1ad81acb1d2f13089356316b4aa9c: Status 404 returned error can't find the container with id f33c774e3734db4bc8e32c4f1c7baa65f3a1ad81acb1d2f13089356316b4aa9c
Sep 16 18:56:32.804402 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:32.804372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm" event={"ID":"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d","Type":"ContainerStarted","Data":"6c4ce4b0a3d9c84506880fb5dfa2a7291971ea1c058e6688b49a9cf2738ccc3e"}
Sep 16 18:56:32.804756 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:32.804408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm" event={"ID":"cb38dfd2-ec66-4fa9-b025-bb4cc22f967d","Type":"ContainerStarted","Data":"f33c774e3734db4bc8e32c4f1c7baa65f3a1ad81acb1d2f13089356316b4aa9c"}
Sep 16 18:56:32.824824 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:32.824779    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm" podStartSLOduration=1.612010452 podStartE2EDuration="1.824760976s" totalImagesPullingTime="212.750524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:32.824120158 +0000 UTC m=+1025.192176078" watchObservedRunningTime="2026-09-16 18:56:32.824760976 +0000 UTC m=+1025.192816878"
Sep 16 18:56:41.792818 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:41.792785    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:41.793330 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:41.792852    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:41.805385 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:41.805357    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:56:41.853613 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:56:41.853585    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-czfjm"
Sep 16 18:57:36.809713 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:36.809681    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:37.407769 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.407733    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66c9cbff69-zkz6l"]
Sep 16 18:57:37.411058 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.411035    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66c9cbff69-zkz6l"
Sep 16 18:57:37.423548 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.423525    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66c9cbff69-zkz6l"]
Sep 16 18:57:37.500981 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.500945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlj6f\" (UniqueName: \"kubernetes.io/projected/840523ce-7791-4c2e-8601-99006dd1df9a-kube-api-access-hlj6f\") pod \"authorino-66c9cbff69-zkz6l\" (UID: \"840523ce-7791-4c2e-8601-99006dd1df9a\") " pod="kuadrant-system/authorino-66c9cbff69-zkz6l"
Sep 16 18:57:37.501143 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.501022    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/840523ce-7791-4c2e-8601-99006dd1df9a-tls-cert\") pod \"authorino-66c9cbff69-zkz6l\" (UID: \"840523ce-7791-4c2e-8601-99006dd1df9a\") " pod="kuadrant-system/authorino-66c9cbff69-zkz6l"
Sep 16 18:57:37.602196 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.602153    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlj6f\" (UniqueName: \"kubernetes.io/projected/840523ce-7791-4c2e-8601-99006dd1df9a-kube-api-access-hlj6f\") pod \"authorino-66c9cbff69-zkz6l\" (UID: \"840523ce-7791-4c2e-8601-99006dd1df9a\") " pod="kuadrant-system/authorino-66c9cbff69-zkz6l"
Sep 16 18:57:37.602395 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.602232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/840523ce-7791-4c2e-8601-99006dd1df9a-tls-cert\") pod \"authorino-66c9cbff69-zkz6l\" (UID: \"840523ce-7791-4c2e-8601-99006dd1df9a\") " pod="kuadrant-system/authorino-66c9cbff69-zkz6l"
Sep 16 18:57:37.604678 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.604650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/840523ce-7791-4c2e-8601-99006dd1df9a-tls-cert\") pod \"authorino-66c9cbff69-zkz6l\" (UID: \"840523ce-7791-4c2e-8601-99006dd1df9a\") " pod="kuadrant-system/authorino-66c9cbff69-zkz6l"
Sep 16 18:57:37.613159 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.613136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlj6f\" (UniqueName: \"kubernetes.io/projected/840523ce-7791-4c2e-8601-99006dd1df9a-kube-api-access-hlj6f\") pod \"authorino-66c9cbff69-zkz6l\" (UID: \"840523ce-7791-4c2e-8601-99006dd1df9a\") " pod="kuadrant-system/authorino-66c9cbff69-zkz6l"
Sep 16 18:57:37.720364 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.720336    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66c9cbff69-zkz6l"
Sep 16 18:57:37.996366 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:37.996343    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66c9cbff69-zkz6l"]
Sep 16 18:57:37.998894 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:57:37.998859    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod840523ce_7791_4c2e_8601_99006dd1df9a.slice/crio-89783857f4c9be30ebaa81520e82e9287dd24850ed3179df514b8e666059a35b WatchSource:0}: Error finding container 89783857f4c9be30ebaa81520e82e9287dd24850ed3179df514b8e666059a35b: Status 404 returned error can't find the container with id 89783857f4c9be30ebaa81520e82e9287dd24850ed3179df514b8e666059a35b
Sep 16 18:57:38.074082 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:38.074051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66c9cbff69-zkz6l" event={"ID":"840523ce-7791-4c2e-8601-99006dd1df9a","Type":"ContainerStarted","Data":"89783857f4c9be30ebaa81520e82e9287dd24850ed3179df514b8e666059a35b"}
Sep 16 18:57:40.083543 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.083496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66c9cbff69-zkz6l" event={"ID":"840523ce-7791-4c2e-8601-99006dd1df9a","Type":"ContainerStarted","Data":"7db8bd28ad6123ff4cc92746853c58a491f74c2f5db20a47629531b0543422b6"}
Sep 16 18:57:40.113613 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.113541    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66c9cbff69-zkz6l" podStartSLOduration=2.074234192 podStartE2EDuration="3.113520145s" totalImagesPullingTime="1.039285953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:40.111377121 +0000 UTC m=+1092.479433024" watchObservedRunningTime="2026-09-16 18:57:40.113520145 +0000 UTC m=+1092.481576047"
Sep 16 18:57:40.165681 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.165642    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85bdf675cb-2lfww"]
Sep 16 18:57:40.165898 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.165865    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85bdf675cb-2lfww" podUID="051c485a-1e7b-4476-b7f7-7f29852d4124" containerName="authorino" containerID="cri-o://d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3" gracePeriod=30
Sep 16 18:57:40.421597 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.421562    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:57:40.528720 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.528682    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/051c485a-1e7b-4476-b7f7-7f29852d4124-tls-cert\") pod \"051c485a-1e7b-4476-b7f7-7f29852d4124\" (UID: \"051c485a-1e7b-4476-b7f7-7f29852d4124\") "
Sep 16 18:57:40.528901 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.528755    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcngx\" (UniqueName: \"kubernetes.io/projected/051c485a-1e7b-4476-b7f7-7f29852d4124-kube-api-access-tcngx\") pod \"051c485a-1e7b-4476-b7f7-7f29852d4124\" (UID: \"051c485a-1e7b-4476-b7f7-7f29852d4124\") "
Sep 16 18:57:40.530832 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.530793    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051c485a-1e7b-4476-b7f7-7f29852d4124-kube-api-access-tcngx" (OuterVolumeSpecName: "kube-api-access-tcngx") pod "051c485a-1e7b-4476-b7f7-7f29852d4124" (UID: "051c485a-1e7b-4476-b7f7-7f29852d4124"). InnerVolumeSpecName "kube-api-access-tcngx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:40.538120 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.538090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051c485a-1e7b-4476-b7f7-7f29852d4124-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "051c485a-1e7b-4476-b7f7-7f29852d4124" (UID: "051c485a-1e7b-4476-b7f7-7f29852d4124"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:40.629523 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.629489    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcngx\" (UniqueName: \"kubernetes.io/projected/051c485a-1e7b-4476-b7f7-7f29852d4124-kube-api-access-tcngx\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:57:40.629523 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.629518    2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/051c485a-1e7b-4476-b7f7-7f29852d4124-tls-cert\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:57:40.847036 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.846994    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:40.942558 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:40.942515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:41.089148 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.089114    2559 generic.go:356] "Generic (PLEG): container finished" podID="051c485a-1e7b-4476-b7f7-7f29852d4124" containerID="d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3" exitCode=0
Sep 16 18:57:41.089571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.089161    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85bdf675cb-2lfww"
Sep 16 18:57:41.089571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.089179    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85bdf675cb-2lfww" event={"ID":"051c485a-1e7b-4476-b7f7-7f29852d4124","Type":"ContainerDied","Data":"d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3"}
Sep 16 18:57:41.089571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.089224    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85bdf675cb-2lfww" event={"ID":"051c485a-1e7b-4476-b7f7-7f29852d4124","Type":"ContainerDied","Data":"5632e22e330e00781e7fa6168941550d1fb9cf211ca948fe4efce594193db552"}
Sep 16 18:57:41.089571 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.089244    2559 scope.go:117] "RemoveContainer" containerID="d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3"
Sep 16 18:57:41.098098 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.098080    2559 scope.go:117] "RemoveContainer" containerID="d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3"
Sep 16 18:57:41.098389 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:57:41.098372    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3\": container with ID starting with d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3 not found: ID does not exist" containerID="d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3"
Sep 16 18:57:41.098438 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.098397    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3"} err="failed to get container status \"d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3\": rpc error: code = NotFound desc = could not find container \"d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3\": container with ID starting with d911cc457bd5ce22f7d53d9cef4fc6b28487525c750130623f8a3b7bb6111ca3 not found: ID does not exist"
Sep 16 18:57:41.125954 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.125929    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85bdf675cb-2lfww"]
Sep 16 18:57:41.133516 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:41.133494    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85bdf675cb-2lfww"]
Sep 16 18:57:42.095745 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:42.095708    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:42.189946 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:42.189913    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:42.221456 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:57:42.221420    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="051c485a-1e7b-4476-b7f7-7f29852d4124" path="/var/lib/kubelet/pods/051c485a-1e7b-4476-b7f7-7f29852d4124/volumes"
Sep 16 18:58:12.192708 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:12.192673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:13.359435 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:13.359401    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:13.750478 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:13.750448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:13.852125 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:13.852083    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:14.186128 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:14.186040    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:16.498917 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:16.498881    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:16.612870 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:16.612823    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:16.720642 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:16.720612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:16.821746 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:16.821674    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:18.107160 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:18.107126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:18.224849 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:18.224813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:18.390881 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:18.390796    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:18.893411 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:18.893382    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:19.011338 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.011310    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:19.041404 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.041372    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"]
Sep 16 18:58:19.041688 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.041677    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="051c485a-1e7b-4476-b7f7-7f29852d4124" containerName="authorino"
Sep 16 18:58:19.041729 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.041690    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="051c485a-1e7b-4476-b7f7-7f29852d4124" containerName="authorino"
Sep 16 18:58:19.041767 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.041742    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="051c485a-1e7b-4476-b7f7-7f29852d4124" containerName="authorino"
Sep 16 18:58:19.044667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.044651    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:19.048850 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.048832    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b1857c\""
Sep 16 18:58:19.048999 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.048864    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b1857c-dockercfg-78jtd\""
Sep 16 18:58:19.054071 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.054049    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"]
Sep 16 18:58:19.111870 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.111841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbk2\" (UniqueName: \"kubernetes.io/projected/4c3257b6-23d8-459b-9632-3c15f1efc165-kube-api-access-wzbk2\") pod \"payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2\" (UID: \"4c3257b6-23d8-459b-9632-3c15f1efc165\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:19.112245 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.111887    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c3257b6-23d8-459b-9632-3c15f1efc165-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2\" (UID: \"4c3257b6-23d8-459b-9632-3c15f1efc165\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:19.122628 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.122594    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"]
Sep 16 18:58:19.125816 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.125796    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:19.138963 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.138940    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"]
Sep 16 18:58:19.212409 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.212377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c3257b6-23d8-459b-9632-3c15f1efc165-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2\" (UID: \"4c3257b6-23d8-459b-9632-3c15f1efc165\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:19.212578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.212443    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef377364-378f-45b9-be7c-b2fc66e57581-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm\" (UID: \"ef377364-378f-45b9-be7c-b2fc66e57581\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:19.212578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.212461    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbk2\" (UniqueName: \"kubernetes.io/projected/4c3257b6-23d8-459b-9632-3c15f1efc165-kube-api-access-wzbk2\") pod \"payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2\" (UID: \"4c3257b6-23d8-459b-9632-3c15f1efc165\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:19.212578 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.212477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhm9f\" (UniqueName: \"kubernetes.io/projected/ef377364-378f-45b9-be7c-b2fc66e57581-kube-api-access-qhm9f\") pod \"payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm\" (UID: \"ef377364-378f-45b9-be7c-b2fc66e57581\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:19.213204 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.213168    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c3257b6-23d8-459b-9632-3c15f1efc165-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2\" (UID: \"4c3257b6-23d8-459b-9632-3c15f1efc165\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:19.227425 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.227395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbk2\" (UniqueName: \"kubernetes.io/projected/4c3257b6-23d8-459b-9632-3c15f1efc165-kube-api-access-wzbk2\") pod \"payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2\" (UID: \"4c3257b6-23d8-459b-9632-3c15f1efc165\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:19.313130 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.313087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef377364-378f-45b9-be7c-b2fc66e57581-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm\" (UID: \"ef377364-378f-45b9-be7c-b2fc66e57581\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:19.313298 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.313142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhm9f\" (UniqueName: \"kubernetes.io/projected/ef377364-378f-45b9-be7c-b2fc66e57581-kube-api-access-qhm9f\") pod \"payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm\" (UID: \"ef377364-378f-45b9-be7c-b2fc66e57581\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:19.313963 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.313942    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef377364-378f-45b9-be7c-b2fc66e57581-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm\" (UID: \"ef377364-378f-45b9-be7c-b2fc66e57581\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:19.327943 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.327917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhm9f\" (UniqueName: \"kubernetes.io/projected/ef377364-378f-45b9-be7c-b2fc66e57581-kube-api-access-qhm9f\") pod \"payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm\" (UID: \"ef377364-378f-45b9-be7c-b2fc66e57581\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:19.354775 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.354748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:19.435649 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.435616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:19.546992 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.546938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:19.635012 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.634966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"]
Sep 16 18:58:19.638274 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:58:19.638242    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3257b6_23d8_459b_9632_3c15f1efc165.slice/crio-482009ae45215827c6112023bf0f30a205ec67c9401085ddc9eb17ae70395c73 WatchSource:0}: Error finding container 482009ae45215827c6112023bf0f30a205ec67c9401085ddc9eb17ae70395c73: Status 404 returned error can't find the container with id 482009ae45215827c6112023bf0f30a205ec67c9401085ddc9eb17ae70395c73
Sep 16 18:58:19.641017 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.640995    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:19.715205 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.715182    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"]
Sep 16 18:58:19.717782 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:58:19.717759    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef377364_378f_45b9_be7c_b2fc66e57581.slice/crio-8fd1378b778b12385dec8780c52ae95236eff4a2f057b7a315b5d6b56ce6a288 WatchSource:0}: Error finding container 8fd1378b778b12385dec8780c52ae95236eff4a2f057b7a315b5d6b56ce6a288: Status 404 returned error can't find the container with id 8fd1378b778b12385dec8780c52ae95236eff4a2f057b7a315b5d6b56ce6a288
Sep 16 18:58:19.736840 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:19.736793    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:20.174884 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.174788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:20.252204 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.252170    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2" event={"ID":"4c3257b6-23d8-459b-9632-3c15f1efc165","Type":"ContainerStarted","Data":"d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648"}
Sep 16 18:58:20.252384 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.252235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2" event={"ID":"4c3257b6-23d8-459b-9632-3c15f1efc165","Type":"ContainerStarted","Data":"482009ae45215827c6112023bf0f30a205ec67c9401085ddc9eb17ae70395c73"}
Sep 16 18:58:20.252384 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.252299    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:20.253811 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.253786    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm" event={"ID":"ef377364-378f-45b9-be7c-b2fc66e57581","Type":"ContainerStarted","Data":"906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088"}
Sep 16 18:58:20.253922 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.253828    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm" event={"ID":"ef377364-378f-45b9-be7c-b2fc66e57581","Type":"ContainerStarted","Data":"8fd1378b778b12385dec8780c52ae95236eff4a2f057b7a315b5d6b56ce6a288"}
Sep 16 18:58:20.253999 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.253921    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:20.291405 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.291347    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2" podStartSLOduration=1.291331274 podStartE2EDuration="1.291331274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:20.289667351 +0000 UTC m=+1132.657723298" watchObservedRunningTime="2026-09-16 18:58:20.291331274 +0000 UTC m=+1132.659387175"
Sep 16 18:58:20.330518 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.330468    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm" podStartSLOduration=1.330453337 podStartE2EDuration="1.330453337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:20.329604939 +0000 UTC m=+1132.697660861" watchObservedRunningTime="2026-09-16 18:58:20.330453337 +0000 UTC m=+1132.698509238"
Sep 16 18:58:20.371671 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.371640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:20.392587 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.392551    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"]
Sep 16 18:58:20.395833 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.395816    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:20.398443 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.398422    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ec32e7-dockercfg-h9k2v\""
Sep 16 18:58:20.399139 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.399122    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ec32e7\""
Sep 16 18:58:20.415315 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.415290    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"]
Sep 16 18:58:20.420592 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.420567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lbzz\" (UniqueName: \"kubernetes.io/projected/4e388f47-3047-42a6-a93f-b84b3288a584-kube-api-access-5lbzz\") pod \"payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn\" (UID: \"4e388f47-3047-42a6-a93f-b84b3288a584\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:20.420681 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.420649    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e388f47-3047-42a6-a93f-b84b3288a584-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn\" (UID: \"4e388f47-3047-42a6-a93f-b84b3288a584\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:20.449253 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.449172    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"]
Sep 16 18:58:20.452659 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.452644    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:20.474439 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.474413    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"]
Sep 16 18:58:20.521623 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.521579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e388f47-3047-42a6-a93f-b84b3288a584-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn\" (UID: \"4e388f47-3047-42a6-a93f-b84b3288a584\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:20.521820 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.521670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lbzz\" (UniqueName: \"kubernetes.io/projected/4e388f47-3047-42a6-a93f-b84b3288a584-kube-api-access-5lbzz\") pod \"payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn\" (UID: \"4e388f47-3047-42a6-a93f-b84b3288a584\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:20.521820 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.521708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk9qv\" (UniqueName: \"kubernetes.io/projected/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-kube-api-access-kk9qv\") pod \"payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml\" (UID: \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:20.521820 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.521747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml\" (UID: \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:20.522350 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.522331    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e388f47-3047-42a6-a93f-b84b3288a584-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn\" (UID: \"4e388f47-3047-42a6-a93f-b84b3288a584\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:20.536157 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.536122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lbzz\" (UniqueName: \"kubernetes.io/projected/4e388f47-3047-42a6-a93f-b84b3288a584-kube-api-access-5lbzz\") pod \"payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn\" (UID: \"4e388f47-3047-42a6-a93f-b84b3288a584\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:20.623245 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.623207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk9qv\" (UniqueName: \"kubernetes.io/projected/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-kube-api-access-kk9qv\") pod \"payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml\" (UID: \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:20.623245 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.623255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml\" (UID: \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:20.623907 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.623887    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml\" (UID: \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:20.633374 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.633348    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk9qv\" (UniqueName: \"kubernetes.io/projected/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-kube-api-access-kk9qv\") pod \"payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml\" (UID: \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:20.706689 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.706656    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:20.764415 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.763512    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:20.812194 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.812162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:20.982716 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:20.982686    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"]
Sep 16 18:58:20.984666 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:58:20.984636    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e388f47_3047_42a6_a93f_b84b3288a584.slice/crio-7716a2cabd5bb58c877f3a86c02529a8fdb9fae5eb616507732e1d446f1d3f86 WatchSource:0}: Error finding container 7716a2cabd5bb58c877f3a86c02529a8fdb9fae5eb616507732e1d446f1d3f86: Status 404 returned error can't find the container with id 7716a2cabd5bb58c877f3a86c02529a8fdb9fae5eb616507732e1d446f1d3f86
Sep 16 18:58:21.046506 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.046480    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"]
Sep 16 18:58:21.050019 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:58:21.049988    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ff2a40_eaef_43db_90dd_3cd36fb512bf.slice/crio-fc5328e98ef6eb3f175c222f091cbede733751f71db5b34dd2f7a668fbeed228 WatchSource:0}: Error finding container fc5328e98ef6eb3f175c222f091cbede733751f71db5b34dd2f7a668fbeed228: Status 404 returned error can't find the container with id fc5328e98ef6eb3f175c222f091cbede733751f71db5b34dd2f7a668fbeed228
Sep 16 18:58:21.261186 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.261095    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml" event={"ID":"d2ff2a40-eaef-43db-90dd-3cd36fb512bf","Type":"ContainerStarted","Data":"7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179"}
Sep 16 18:58:21.261186 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.261168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml" event={"ID":"d2ff2a40-eaef-43db-90dd-3cd36fb512bf","Type":"ContainerStarted","Data":"fc5328e98ef6eb3f175c222f091cbede733751f71db5b34dd2f7a668fbeed228"}
Sep 16 18:58:21.261683 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.261204    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:21.263357 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.263328    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn" event={"ID":"4e388f47-3047-42a6-a93f-b84b3288a584","Type":"ContainerStarted","Data":"f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2"}
Sep 16 18:58:21.263506 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.263379    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn" event={"ID":"4e388f47-3047-42a6-a93f-b84b3288a584","Type":"ContainerStarted","Data":"7716a2cabd5bb58c877f3a86c02529a8fdb9fae5eb616507732e1d446f1d3f86"}
Sep 16 18:58:21.263506 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.263492    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:21.289622 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.289581    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml" podStartSLOduration=1.289567654 podStartE2EDuration="1.289567654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:21.287059777 +0000 UTC m=+1133.655115697" watchObservedRunningTime="2026-09-16 18:58:21.289567654 +0000 UTC m=+1133.657623545"
Sep 16 18:58:21.308390 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.308344    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn" podStartSLOduration=1.308331285 podStartE2EDuration="1.308331285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:21.305718317 +0000 UTC m=+1133.673774231" watchObservedRunningTime="2026-09-16 18:58:21.308331285 +0000 UTC m=+1133.676387180"
Sep 16 18:58:21.751585 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.751550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:21.850686 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:21.850651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.180172 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.180099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.296144 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.296113    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.545414 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.545379    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.774062 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.774026    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.868301 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.868213    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:22.891393 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.891362    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"]
Sep 16 18:58:22.894701 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.894685    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:22.897839 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.897820    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c090c8\""
Sep 16 18:58:22.910003 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.909961    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"]
Sep 16 18:58:22.947826 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.947796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs\" (UID: \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:22.947983 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:22.947844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4b8\" (UniqueName: \"kubernetes.io/projected/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-kube-api-access-tn4b8\") pod \"payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs\" (UID: \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:23.049169 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:23.049132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs\" (UID: \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:23.049328 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:23.049202    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4b8\" (UniqueName: \"kubernetes.io/projected/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-kube-api-access-tn4b8\") pod \"payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs\" (UID: \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:23.049833 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:23.049815    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs\" (UID: \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:23.060357 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:23.060336    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4b8\" (UniqueName: \"kubernetes.io/projected/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-kube-api-access-tn4b8\") pod \"payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs\" (UID: \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:23.204660 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:23.204567    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:23.335759 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:23.335724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:23.504118 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:23.504083    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"]
Sep 16 18:58:23.507404 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:58:23.507376    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eda8e0f_ae0f_49d7_aa61_3d3c82d43f9e.slice/crio-b52766b5d36071046ad0d45480c48049349d87ffcb20750b051ef2b4e8a3c2cf WatchSource:0}: Error finding container b52766b5d36071046ad0d45480c48049349d87ffcb20750b051ef2b4e8a3c2cf: Status 404 returned error can't find the container with id b52766b5d36071046ad0d45480c48049349d87ffcb20750b051ef2b4e8a3c2cf
Sep 16 18:58:24.278436 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:24.278397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs" event={"ID":"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e","Type":"ContainerStarted","Data":"98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2"}
Sep 16 18:58:24.278621 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:24.278466    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs" event={"ID":"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e","Type":"ContainerStarted","Data":"b52766b5d36071046ad0d45480c48049349d87ffcb20750b051ef2b4e8a3c2cf"}
Sep 16 18:58:24.278621 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:24.278520    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:24.320887 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:24.320822    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs" podStartSLOduration=2.320801386 podStartE2EDuration="2.320801386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:24.319735759 +0000 UTC m=+1136.687791679" watchObservedRunningTime="2026-09-16 18:58:24.320801386 +0000 UTC m=+1136.688857288"
Sep 16 18:58:24.444451 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:24.444409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:24.537459 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:24.537367    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:24.671325 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:24.671293    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:24.952440 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:24.952368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:25.049508 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:25.049475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:25.154541 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:25.154507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:25.568273 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:25.568240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:25.661534 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:25.661502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:26.018029 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:26.017961    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:27.220680 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:27.220653    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:27.320019 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:27.319968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:27.474940 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:27.474762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:27.576949 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:27.576920    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:27.691800 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:27.691767    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:27.789336 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:27.789262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:27.909564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:27.909527    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:28.051996 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.051901    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"]
Sep 16 18:58:28.055604 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.055341    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:28.058441 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.058416    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-2efa6f-dockercfg-krxx2\""
Sep 16 18:58:28.067035 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.067016    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-2efa6f\""
Sep 16 18:58:28.081569 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.081539    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"]
Sep 16 18:58:28.090347 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.090324    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"]
Sep 16 18:58:28.091840 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.091815    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk\" (UID: \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:28.091933 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.091885    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7tt\" (UniqueName: \"kubernetes.io/projected/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-kube-api-access-vn7tt\") pod \"payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk\" (UID: \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:28.094633 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.094613    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:28.119774 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.119747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"]
Sep 16 18:58:28.193085 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.193052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0478f39-b092-49ad-8576-f581937b3669-plugins-config-volume\") pod \"payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k\" (UID: \"d0478f39-b092-49ad-8576-f581937b3669\") " pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:28.193229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.193103    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk\" (UID: \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:28.193229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.193172    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7tt\" (UniqueName: \"kubernetes.io/projected/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-kube-api-access-vn7tt\") pod \"payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk\" (UID: \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:28.193229 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.193197    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vcjw\" (UniqueName: \"kubernetes.io/projected/d0478f39-b092-49ad-8576-f581937b3669-kube-api-access-2vcjw\") pod \"payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k\" (UID: \"d0478f39-b092-49ad-8576-f581937b3669\") " pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:28.193823 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.193803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk\" (UID: \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:28.212919 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.212893    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7tt\" (UniqueName: \"kubernetes.io/projected/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-kube-api-access-vn7tt\") pod \"payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk\" (UID: \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:28.294027 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.293998    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0478f39-b092-49ad-8576-f581937b3669-plugins-config-volume\") pod \"payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k\" (UID: \"d0478f39-b092-49ad-8576-f581937b3669\") " pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:28.294456 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.294118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vcjw\" (UniqueName: \"kubernetes.io/projected/d0478f39-b092-49ad-8576-f581937b3669-kube-api-access-2vcjw\") pod \"payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k\" (UID: \"d0478f39-b092-49ad-8576-f581937b3669\") " pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:28.294785 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.294766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0478f39-b092-49ad-8576-f581937b3669-plugins-config-volume\") pod \"payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k\" (UID: \"d0478f39-b092-49ad-8576-f581937b3669\") " pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:28.311050 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.310994    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vcjw\" (UniqueName: \"kubernetes.io/projected/d0478f39-b092-49ad-8576-f581937b3669-kube-api-access-2vcjw\") pod \"payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k\" (UID: \"d0478f39-b092-49ad-8576-f581937b3669\") " pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:28.364804 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.364773    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:28.403558 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.403530    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:28.542152 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.542122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:28.646000 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.645624    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"]
Sep 16 18:58:28.708453 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:28.708422    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"]
Sep 16 18:58:28.708816 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:58:28.708795    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0478f39_b092_49ad_8576_f581937b3669.slice/crio-65b4efdd633ea59e91511fe5e9ccb8caf92a5d5322875203027ba3a79b56d142 WatchSource:0}: Error finding container 65b4efdd633ea59e91511fe5e9ccb8caf92a5d5322875203027ba3a79b56d142: Status 404 returned error can't find the container with id 65b4efdd633ea59e91511fe5e9ccb8caf92a5d5322875203027ba3a79b56d142
Sep 16 18:58:29.303397 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.303363    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk" event={"ID":"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146","Type":"ContainerStarted","Data":"1d6a05d37804be038892ae4a1083b3903ba5f7181871b1176004688a1b2d02df"}
Sep 16 18:58:29.303855 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.303423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk" event={"ID":"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146","Type":"ContainerStarted","Data":"a23907c73d8acfa3f1ac49d8a110af73f28a6a34746a31b26a8fb00adbc65ea3"}
Sep 16 18:58:29.303855 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.303456    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:29.304894 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.304868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k" event={"ID":"d0478f39-b092-49ad-8576-f581937b3669","Type":"ContainerStarted","Data":"ae2ab5c05456de5e2bc7a915d30278684aab49e79230d7e7dd8c77aff47c6fb3"}
Sep 16 18:58:29.305009 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.304914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k" event={"ID":"d0478f39-b092-49ad-8576-f581937b3669","Type":"ContainerStarted","Data":"65b4efdd633ea59e91511fe5e9ccb8caf92a5d5322875203027ba3a79b56d142"}
Sep 16 18:58:29.305009 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.304953    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:29.325552 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.325509    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk" podStartSLOduration=1.3254940720000001 podStartE2EDuration="1.325494072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:29.324648926 +0000 UTC m=+1141.692704845" watchObservedRunningTime="2026-09-16 18:58:29.325494072 +0000 UTC m=+1141.693550008"
Sep 16 18:58:29.343256 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.343213    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k" podStartSLOduration=1.343199027 podStartE2EDuration="1.343199027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:29.342169288 +0000 UTC m=+1141.710225224" watchObservedRunningTime="2026-09-16 18:58:29.343199027 +0000 UTC m=+1141.711254927"
Sep 16 18:58:29.649765 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:29.649662    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:30.606719 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:30.606682    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:30.707004 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:30.706939    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:30.963064 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:30.963022    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:31.234820 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.234724    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:31.264922 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.264899    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 18:58:31.265077 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.264958    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 18:58:31.402346 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.402313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:31.511670 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.511586    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:31.616950 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.616916    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"]
Sep 16 18:58:31.620233 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.620217    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:31.629687 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.629667    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4300c3c2\""
Sep 16 18:58:31.629809 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.629793    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4300c3c2-dockercfg-v9s55\""
Sep 16 18:58:31.645738 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.645711    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"]
Sep 16 18:58:31.721677 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.721645    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tgng\" (UniqueName: \"kubernetes.io/projected/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-kube-api-access-9tgng\") pod \"payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw\" (UID: \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\") " pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:31.721812 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.721691    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-plugins-config-volume\") pod \"payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw\" (UID: \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\") " pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:31.758521 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.758474    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:31.822811 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.822740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tgng\" (UniqueName: \"kubernetes.io/projected/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-kube-api-access-9tgng\") pod \"payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw\" (UID: \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\") " pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:31.822811 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.822783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-plugins-config-volume\") pod \"payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw\" (UID: \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\") " pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:31.823448 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.823429    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-plugins-config-volume\") pod \"payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw\" (UID: \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\") " pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:31.837252 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.837232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tgng\" (UniqueName: \"kubernetes.io/projected/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-kube-api-access-9tgng\") pod \"payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw\" (UID: \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\") " pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:31.857813 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.857788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:31.930521 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.930492    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:31.985126 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:31.985086    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:32.223067 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:58:32.223026    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986ff851_87ca_4b06_9ac3_ba4d6ec16a7c.slice/crio-901c02d8a602e885d2946be0ea8122e9ce2ed57bd18b92c1fab31305b72dc6c7 WatchSource:0}: Error finding container 901c02d8a602e885d2946be0ea8122e9ce2ed57bd18b92c1fab31305b72dc6c7: Status 404 returned error can't find the container with id 901c02d8a602e885d2946be0ea8122e9ce2ed57bd18b92c1fab31305b72dc6c7
Sep 16 18:58:32.223834 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:32.223808    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"]
Sep 16 18:58:32.270477 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:32.270452    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 18:58:32.270648 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:32.270624    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 18:58:32.305377 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:32.305348    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:32.320424 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:32.320393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw" event={"ID":"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c","Type":"ContainerStarted","Data":"f5db63d712b54d679dbc4b074dbd584554b2dd517ec2a45766734f31732244ca"}
Sep 16 18:58:32.320564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:32.320448    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:32.320564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:32.320472    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw" event={"ID":"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c","Type":"ContainerStarted","Data":"901c02d8a602e885d2946be0ea8122e9ce2ed57bd18b92c1fab31305b72dc6c7"}
Sep 16 18:58:32.356232 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:32.356177    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw" podStartSLOduration=1.356160558 podStartE2EDuration="1.356160558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:32.353660109 +0000 UTC m=+1144.721716032" watchObservedRunningTime="2026-09-16 18:58:32.356160558 +0000 UTC m=+1144.724216458"
Sep 16 18:58:33.505688 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:33.505648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:34.589603 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:34.589567    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.285012 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:35.284966    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 18:58:35.922705 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:35.922670    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.049033 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:36.049000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.222494 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:36.222463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.389282 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:36.389251    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.509963 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:36.509897    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.623551 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:36.623516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.843665 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:36.843588    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:36.976367 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:36.976335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:37.115673 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:37.115597    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:37.246513 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:37.246482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:37.411652 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:37.411579    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:37.884747 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:37.884719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:37.985557 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:37.985529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:38.092778 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:38.092744    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:38.307522 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:38.307494    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:39.146546 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:39.146512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:39.444632 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:39.444559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:39.898532 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:39.898500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:40.311778 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:40.311750    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 18:58:40.312166 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:40.311820    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 18:58:40.445501 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:40.445466    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:41.415812 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:41.415769    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:41.522256 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:41.522224    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:41.611346 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:41.611318    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:42.974140 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:42.974106    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:43.328389 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:43.328311    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:44.019520 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:44.019489    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:44.928997 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:44.928937    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:45.352036 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:45.351999    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:46.005031 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:46.005003    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:46.105193 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:46.105158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:46.477455 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:46.477426    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:46.613553 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:46.613518    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:46.745591 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:46.745517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:46.841037 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:46.840999    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:46.928180 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:46.928151    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:47.455647 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:47.455615    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:47.550667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:47.550629    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:48.455714 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:48.455674    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:49.321162 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:49.321130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:50.225476 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:50.225397    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:51.321741 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:51.321707    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:52.349498 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:52.349471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:53.154426 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.154399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:53.187433 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.187400    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"]
Sep 16 18:58:53.187671 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.187647    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw" podUID="986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" containerName="payload-processing" containerID="cri-o://f5db63d712b54d679dbc4b074dbd584554b2dd517ec2a45766734f31732244ca" gracePeriod=30
Sep 16 18:58:53.245324 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.245296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:53.327274 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.327230    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw" podUID="986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.37:9004: connect: connection refused"
Sep 16 18:58:53.418842 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.418810    2559 generic.go:356] "Generic (PLEG): container finished" podID="986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" containerID="f5db63d712b54d679dbc4b074dbd584554b2dd517ec2a45766734f31732244ca" exitCode=0
Sep 16 18:58:53.419183 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.418846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw" event={"ID":"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c","Type":"ContainerDied","Data":"f5db63d712b54d679dbc4b074dbd584554b2dd517ec2a45766734f31732244ca"}
Sep 16 18:58:53.419183 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.418899    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw" event={"ID":"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c","Type":"ContainerDied","Data":"901c02d8a602e885d2946be0ea8122e9ce2ed57bd18b92c1fab31305b72dc6c7"}
Sep 16 18:58:53.419183 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.418921    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="901c02d8a602e885d2946be0ea8122e9ce2ed57bd18b92c1fab31305b72dc6c7"
Sep 16 18:58:53.429292 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.429266    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:53.505564 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.505535    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-plugins-config-volume\") pod \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\" (UID: \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\") "
Sep 16 18:58:53.505759 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.505604    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tgng\" (UniqueName: \"kubernetes.io/projected/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-kube-api-access-9tgng\") pod \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\" (UID: \"986ff851-87ca-4b06-9ac3-ba4d6ec16a7c\") "
Sep 16 18:58:53.505904 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.505882    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" (UID: "986ff851-87ca-4b06-9ac3-ba4d6ec16a7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:53.507644 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.507615    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-kube-api-access-9tgng" (OuterVolumeSpecName: "kube-api-access-9tgng") pod "986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" (UID: "986ff851-87ca-4b06-9ac3-ba4d6ec16a7c"). InnerVolumeSpecName "kube-api-access-9tgng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:53.607101 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.607069    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tgng\" (UniqueName: \"kubernetes.io/projected/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-kube-api-access-9tgng\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:58:53.607101 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.607097    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:58:53.634617 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.634584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:53.735566 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:53.735536    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:54.126178 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:54.126098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:54.236577 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:54.236550    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:54.342081 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:54.342047    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:54.423938 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:54.423861    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"
Sep 16 18:58:54.454140 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:54.454112    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"]
Sep 16 18:58:54.478824 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:54.478796    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4300c3c2-c47dbdc5-6qlxw"]
Sep 16 18:58:54.580928 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:54.580894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:54.723613 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:54.723580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:55.134548 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:55.134462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:55.879938 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:55.879906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:56.108374 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:56.108341    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:56.222443 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:56.222412    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" path="/var/lib/kubelet/pods/986ff851-87ca-4b06-9ac3-ba4d6ec16a7c/volumes"
Sep 16 18:58:56.971406 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:56.971372    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:58.319262 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:58.319226    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:58.419213 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:58.419178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:59.643616 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:58:59.643582    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:00.535396 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:00.535365    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:01.718224 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:01.718190    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:03.114647 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.114612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:03.220015 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.219958    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:03.409623 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.409538    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:03.763195 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.763158    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:03.857634 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.857606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:03.889553 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.889515    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"]
Sep 16 18:59:03.889886 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.889873    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" containerName="payload-processing"
Sep 16 18:59:03.889934 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.889887    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" containerName="payload-processing"
Sep 16 18:59:03.889989 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.889942    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="986ff851-87ca-4b06-9ac3-ba4d6ec16a7c" containerName="payload-processing"
Sep 16 18:59:03.893172 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.893144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:03.896114 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.896094    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-78ad8baa\""
Sep 16 18:59:03.903605 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.903582    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"]
Sep 16 18:59:03.907036 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.907014    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:03.907656 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.907629    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"]
Sep 16 18:59:03.910071 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.910049    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-78ad8baa-dockercfg-q55ql\""
Sep 16 18:59:03.925848 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.925821    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"]
Sep 16 18:59:03.997394 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.997361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9140a289-ae4e-49d5-99b5-693835af2089-plugins-config-volume\") pod \"payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt\" (UID: \"9140a289-ae4e-49d5-99b5-693835af2089\") " pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:03.997601 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.997402    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4\" (UID: \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:03.997601 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.997432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck46p\" (UniqueName: \"kubernetes.io/projected/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-kube-api-access-ck46p\") pod \"payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4\" (UID: \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:03.997601 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:03.997455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6hv9\" (UniqueName: \"kubernetes.io/projected/9140a289-ae4e-49d5-99b5-693835af2089-kube-api-access-k6hv9\") pod \"payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt\" (UID: \"9140a289-ae4e-49d5-99b5-693835af2089\") " pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:04.098775 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.098669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9140a289-ae4e-49d5-99b5-693835af2089-plugins-config-volume\") pod \"payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt\" (UID: \"9140a289-ae4e-49d5-99b5-693835af2089\") " pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:04.098775 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.098717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4\" (UID: \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:04.099051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.098842    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck46p\" (UniqueName: \"kubernetes.io/projected/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-kube-api-access-ck46p\") pod \"payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4\" (UID: \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:04.099051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.098892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6hv9\" (UniqueName: \"kubernetes.io/projected/9140a289-ae4e-49d5-99b5-693835af2089-kube-api-access-k6hv9\") pod \"payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt\" (UID: \"9140a289-ae4e-49d5-99b5-693835af2089\") " pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:04.099513 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.099489    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4\" (UID: \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:04.099513 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.099500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9140a289-ae4e-49d5-99b5-693835af2089-plugins-config-volume\") pod \"payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt\" (UID: \"9140a289-ae4e-49d5-99b5-693835af2089\") " pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:04.111185 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.111161    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6hv9\" (UniqueName: \"kubernetes.io/projected/9140a289-ae4e-49d5-99b5-693835af2089-kube-api-access-k6hv9\") pod \"payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt\" (UID: \"9140a289-ae4e-49d5-99b5-693835af2089\") " pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:04.111291 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.111218    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck46p\" (UniqueName: \"kubernetes.io/projected/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-kube-api-access-ck46p\") pod \"payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4\" (UID: \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:04.203331 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.203286    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:04.217186 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.217159    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:04.419716 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.419640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:04.484651 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.484585    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"]
Sep 16 18:59:04.486990 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:59:04.486945    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ae3623_d333_4dd1_83fc_b5cf1ed4ab5d.slice/crio-dd054cf966461df49fc46c241c90a7496d1d3801673013bd6a2eb49c52f4b697 WatchSource:0}: Error finding container dd054cf966461df49fc46c241c90a7496d1d3801673013bd6a2eb49c52f4b697: Status 404 returned error can't find the container with id dd054cf966461df49fc46c241c90a7496d1d3801673013bd6a2eb49c52f4b697
Sep 16 18:59:04.517629 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:04.517610    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"]
Sep 16 18:59:04.519204 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:59:04.519177    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9140a289_ae4e_49d5_99b5_693835af2089.slice/crio-fe503282fc868b5f115aebc5343d1f2618b42b05c58423b8b894bba390c327d5 WatchSource:0}: Error finding container fe503282fc868b5f115aebc5343d1f2618b42b05c58423b8b894bba390c327d5: Status 404 returned error can't find the container with id fe503282fc868b5f115aebc5343d1f2618b42b05c58423b8b894bba390c327d5
Sep 16 18:59:05.477644 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.477601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4" event={"ID":"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d","Type":"ContainerStarted","Data":"921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1"}
Sep 16 18:59:05.478111 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.477660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4" event={"ID":"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d","Type":"ContainerStarted","Data":"dd054cf966461df49fc46c241c90a7496d1d3801673013bd6a2eb49c52f4b697"}
Sep 16 18:59:05.478111 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.477693    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:05.479701 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.479676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt" event={"ID":"9140a289-ae4e-49d5-99b5-693835af2089","Type":"ContainerStarted","Data":"a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be"}
Sep 16 18:59:05.479816 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.479725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt" event={"ID":"9140a289-ae4e-49d5-99b5-693835af2089","Type":"ContainerStarted","Data":"fe503282fc868b5f115aebc5343d1f2618b42b05c58423b8b894bba390c327d5"}
Sep 16 18:59:05.479816 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.479755    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:05.517366 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.517320    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4" podStartSLOduration=2.517306921 podStartE2EDuration="2.517306921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:05.516543458 +0000 UTC m=+1177.884599378" watchObservedRunningTime="2026-09-16 18:59:05.517306921 +0000 UTC m=+1177.885362822"
Sep 16 18:59:05.541755 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.541710    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt" podStartSLOduration=2.541694931 podStartE2EDuration="2.541694931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:05.541412097 +0000 UTC m=+1177.909468018" watchObservedRunningTime="2026-09-16 18:59:05.541694931 +0000 UTC m=+1177.909750832"
Sep 16 18:59:05.643265 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:05.643238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:06.623474 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:06.623432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:07.525154 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:07.525125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:16.486988 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:16.486940    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:16.487489 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:16.487037    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:24.334256 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:24.334218    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:26.564228 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:26.564196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:27.546676 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:27.546633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:28.041397 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:28.041365    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:59:28.041851 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:28.041659    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 18:59:28.600838 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:28.600803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:29.846665 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:29.846634    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:30.922495 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:30.922457    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:32.058417 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:32.058380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:33.294888 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.294854    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:33.328162 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.328132    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"]
Sep 16 18:59:33.331533 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.331516    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.336036 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.336016    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1857c\"/\"simulator-w0-b1857c-kserve-self-signed-certs\""
Sep 16 18:59:33.336154 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.336016    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1857c\"/\"default-dockercfg-9xd2j\""
Sep 16 18:59:33.336154 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.336088    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1857c\"/\"openshift-service-ca.crt\""
Sep 16 18:59:33.336154 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.336134    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1857c\"/\"kube-root-ca.crt\""
Sep 16 18:59:33.357945 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.357916    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"]
Sep 16 18:59:33.437862 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.437830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-home\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.438051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.437869    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tmp-dir\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.438051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.437943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tls-certs\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.438051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.437985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-dshm\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.438182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.438068    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqgms\" (UniqueName: \"kubernetes.io/projected/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-kube-api-access-zqgms\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.438182 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.438125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-model-cache\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.506296 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.506252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:33.538654 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.538631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-model-cache\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.538748 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.538669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-home\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.538748 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.538687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tmp-dir\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.538748 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.538719    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tls-certs\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.538748 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.538738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-dshm\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.538935 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.538766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqgms\" (UniqueName: \"kubernetes.io/projected/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-kube-api-access-zqgms\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.539112 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.539090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-model-cache\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.539191 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.539107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-home\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.539191 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.539167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tmp-dir\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.541091 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.541065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-dshm\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.541291 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.541270    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tls-certs\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.561771 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.561704    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqgms\" (UniqueName: \"kubernetes.io/projected/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-kube-api-access-zqgms\") pod \"simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") " pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.639984 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.639951    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:33.919604 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:33.919577    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"]
Sep 16 18:59:33.920878 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:59:33.920858    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca1b65e7_3f6e_414b_b2e7_01be2b88da4e.slice/crio-a76892b36bf3db36893d73e29913cc87a3e3338adec8c803fd64ab5b7ee45714 WatchSource:0}: Error finding container a76892b36bf3db36893d73e29913cc87a3e3338adec8c803fd64ab5b7ee45714: Status 404 returned error can't find the container with id a76892b36bf3db36893d73e29913cc87a3e3338adec8c803fd64ab5b7ee45714
Sep 16 18:59:34.543944 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:34.543909    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:34.614798 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:34.614754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n" event={"ID":"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e","Type":"ContainerStarted","Data":"5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338"}
Sep 16 18:59:34.614798 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:34.614801    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n" event={"ID":"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e","Type":"ContainerStarted","Data":"a76892b36bf3db36893d73e29913cc87a3e3338adec8c803fd64ab5b7ee45714"}
Sep 16 18:59:34.644127 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:34.644078    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n" podStartSLOduration=1.644061362 podStartE2EDuration="1.644061362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:34.642401295 +0000 UTC m=+1207.010457220" watchObservedRunningTime="2026-09-16 18:59:34.644061362 +0000 UTC m=+1207.012117263"
Sep 16 18:59:35.865665 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:35.865628    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:37.645527 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:37.645489    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:39.809985 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:39.809943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:40.922563 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:40.922517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.032548 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.032513    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"]
Sep 16 18:59:41.032849 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.032797    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt" podUID="9140a289-ae4e-49d5-99b5-693835af2089" containerName="payload-processing" containerID="cri-o://a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be" gracePeriod=30
Sep 16 18:59:41.051563 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.051538    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"]
Sep 16 18:59:41.051757 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.051737    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4" podUID="f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d" containerName="payload-pre-processing" containerID="cri-o://921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1" gracePeriod=30
Sep 16 18:59:41.097311 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.097283    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.204868 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.204784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.324270 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.324245    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:41.327132 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.327111    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:41.410073 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.410044    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9140a289-ae4e-49d5-99b5-693835af2089-plugins-config-volume\") pod \"9140a289-ae4e-49d5-99b5-693835af2089\" (UID: \"9140a289-ae4e-49d5-99b5-693835af2089\") "
Sep 16 18:59:41.410265 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.410103    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-plugins-config-volume\") pod \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\" (UID: \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\") "
Sep 16 18:59:41.410265 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.410137    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6hv9\" (UniqueName: \"kubernetes.io/projected/9140a289-ae4e-49d5-99b5-693835af2089-kube-api-access-k6hv9\") pod \"9140a289-ae4e-49d5-99b5-693835af2089\" (UID: \"9140a289-ae4e-49d5-99b5-693835af2089\") "
Sep 16 18:59:41.410265 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.410228    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck46p\" (UniqueName: \"kubernetes.io/projected/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-kube-api-access-ck46p\") pod \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\" (UID: \"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d\") "
Sep 16 18:59:41.410505 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.410480    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9140a289-ae4e-49d5-99b5-693835af2089-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9140a289-ae4e-49d5-99b5-693835af2089" (UID: "9140a289-ae4e-49d5-99b5-693835af2089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:41.410550 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.410506    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d" (UID: "f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:41.412363 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.412337    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9140a289-ae4e-49d5-99b5-693835af2089-kube-api-access-k6hv9" (OuterVolumeSpecName: "kube-api-access-k6hv9") pod "9140a289-ae4e-49d5-99b5-693835af2089" (UID: "9140a289-ae4e-49d5-99b5-693835af2089"). InnerVolumeSpecName "kube-api-access-k6hv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:41.412470 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.412366    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-kube-api-access-ck46p" (OuterVolumeSpecName: "kube-api-access-ck46p") pod "f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d" (UID: "f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d"). InnerVolumeSpecName "kube-api-access-ck46p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:41.416212 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.416175    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.511667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.511580    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck46p\" (UniqueName: \"kubernetes.io/projected/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-kube-api-access-ck46p\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.511667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.511610    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9140a289-ae4e-49d5-99b5-693835af2089-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.511667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.511622    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.511667 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.511630    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6hv9\" (UniqueName: \"kubernetes.io/projected/9140a289-ae4e-49d5-99b5-693835af2089-kube-api-access-k6hv9\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 18:59:41.513161 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.513137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.649110 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.649072    2559 generic.go:356] "Generic (PLEG): container finished" podID="9140a289-ae4e-49d5-99b5-693835af2089" containerID="a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be" exitCode=0
Sep 16 18:59:41.649300 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.649118    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"
Sep 16 18:59:41.649300 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.649152    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt" event={"ID":"9140a289-ae4e-49d5-99b5-693835af2089","Type":"ContainerDied","Data":"a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be"}
Sep 16 18:59:41.649300 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.649209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt" event={"ID":"9140a289-ae4e-49d5-99b5-693835af2089","Type":"ContainerDied","Data":"fe503282fc868b5f115aebc5343d1f2618b42b05c58423b8b894bba390c327d5"}
Sep 16 18:59:41.649300 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.649239    2559 scope.go:117] "RemoveContainer" containerID="a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be"
Sep 16 18:59:41.650544 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.650526    2559 generic.go:356] "Generic (PLEG): container finished" podID="f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d" containerID="921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1" exitCode=0
Sep 16 18:59:41.650651 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.650575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"
Sep 16 18:59:41.650651 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.650582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4" event={"ID":"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d","Type":"ContainerDied","Data":"921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1"}
Sep 16 18:59:41.650651 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.650618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4" event={"ID":"f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d","Type":"ContainerDied","Data":"dd054cf966461df49fc46c241c90a7496d1d3801673013bd6a2eb49c52f4b697"}
Sep 16 18:59:41.657835 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.657809    2559 scope.go:117] "RemoveContainer" containerID="a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be"
Sep 16 18:59:41.658146 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:59:41.658116    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be\": container with ID starting with a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be not found: ID does not exist" containerID="a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be"
Sep 16 18:59:41.658250 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.658142    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be"} err="failed to get container status \"a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be\": rpc error: code = NotFound desc = could not find container \"a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be\": container with ID starting with a9a1aa9bcefedc4cbf16b09ee0376dc86352e486451980b1bd7537d44be3d1be not found: ID does not exist"
Sep 16 18:59:41.658250 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.658158    2559 scope.go:117] "RemoveContainer" containerID="921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1"
Sep 16 18:59:41.666280 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.666258    2559 scope.go:117] "RemoveContainer" containerID="921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1"
Sep 16 18:59:41.666522 ip-10-0-141-134 kubenswrapper[2559]: E0916 18:59:41.666503    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1\": container with ID starting with 921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1 not found: ID does not exist" containerID="921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1"
Sep 16 18:59:41.666566 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.666531    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1"} err="failed to get container status \"921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1\": rpc error: code = NotFound desc = could not find container \"921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1\": container with ID starting with 921e436952e5f6bbb1310995ee96d2fcc69321437f916618e17f679b19d6d5f1 not found: ID does not exist"
Sep 16 18:59:41.682010 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.681988    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.700116 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.700090    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"]
Sep 16 18:59:41.715416 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.715389    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-78ad8baa-6c55d8f58b-fndb4"]
Sep 16 18:59:41.741049 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.741018    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"]
Sep 16 18:59:41.744252 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.744226    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-78ad8baa-5857db4578-cjbzt"]
Sep 16 18:59:41.942369 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:41.942276    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:42.130321 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.130284    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:42.207666 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.207632    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"]
Sep 16 18:59:42.207999 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.207986    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d" containerName="payload-pre-processing"
Sep 16 18:59:42.208051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.208001    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d" containerName="payload-pre-processing"
Sep 16 18:59:42.208051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.208011    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9140a289-ae4e-49d5-99b5-693835af2089" containerName="payload-processing"
Sep 16 18:59:42.208051 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.208017    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9140a289-ae4e-49d5-99b5-693835af2089" containerName="payload-processing"
Sep 16 18:59:42.208143 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.208084    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9140a289-ae4e-49d5-99b5-693835af2089" containerName="payload-processing"
Sep 16 18:59:42.208143 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.208092    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d" containerName="payload-pre-processing"
Sep 16 18:59:42.212334 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.212318    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.215427 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.215406    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"default-dockercfg-9h2xx\""
Sep 16 18:59:42.215708 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.215693    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"simulator-w4-c090c8-kserve-self-signed-certs\""
Sep 16 18:59:42.216732 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.216711    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"kube-root-ca.crt\""
Sep 16 18:59:42.216850 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.216798    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"openshift-service-ca.crt\""
Sep 16 18:59:42.222599 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.222577    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9140a289-ae4e-49d5-99b5-693835af2089" path="/var/lib/kubelet/pods/9140a289-ae4e-49d5-99b5-693835af2089/volumes"
Sep 16 18:59:42.222902 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.222890    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d" path="/var/lib/kubelet/pods/f9ae3623-d333-4dd1-83fc-b5cf1ed4ab5d/volumes"
Sep 16 18:59:42.230151 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.230129    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"]
Sep 16 18:59:42.316321 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.316289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkv4d\" (UniqueName: \"kubernetes.io/projected/7099c17b-763b-4c64-b8af-3c64de776e64-kube-api-access-wkv4d\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.316321 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.316323    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-tmp-dir\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.316527 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.316360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-model-cache\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.316527 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.316438    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7099c17b-763b-4c64-b8af-3c64de776e64-tls-certs\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.316606 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.316548    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-dshm\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.316672 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.316655    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-home\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.417922 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.417885    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-model-cache\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.417922 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.417923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7099c17b-763b-4c64-b8af-3c64de776e64-tls-certs\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.418171 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.417957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-dshm\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.418171 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.418041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-home\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.418171 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.418099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkv4d\" (UniqueName: \"kubernetes.io/projected/7099c17b-763b-4c64-b8af-3c64de776e64-kube-api-access-wkv4d\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.418171 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.418133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-tmp-dir\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.418388 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.418361    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-model-cache\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.418446 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.418417    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-tmp-dir\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.418504 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.418488    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-home\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.420287 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.420266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-dshm\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.420409 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.420392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7099c17b-763b-4c64-b8af-3c64de776e64-tls-certs\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.434445 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.434420    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkv4d\" (UniqueName: \"kubernetes.io/projected/7099c17b-763b-4c64-b8af-3c64de776e64-kube-api-access-wkv4d\") pod \"simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") " pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.521394 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.521313    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:42.614479 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.614447    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:42.730554 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.730519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:42.832722 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.832631    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"]
Sep 16 18:59:42.835255 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:59:42.835231    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7099c17b_763b_4c64_b8af_3c64de776e64.slice/crio-1466d95e0965488a6d754e2aa0590e690ccc8f1131202edbeee33b736b55e0cc WatchSource:0}: Error finding container 1466d95e0965488a6d754e2aa0590e690ccc8f1131202edbeee33b736b55e0cc: Status 404 returned error can't find the container with id 1466d95e0965488a6d754e2aa0590e690ccc8f1131202edbeee33b736b55e0cc
Sep 16 18:59:42.915475 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:42.915446    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:43.640284 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:43.640237    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:43.640284 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:43.640299    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:43.652619 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:43.652597    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:43.663650 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:43.663618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2" event={"ID":"7099c17b-763b-4c64-b8af-3c64de776e64","Type":"ContainerStarted","Data":"3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d"}
Sep 16 18:59:43.663787 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:43.663658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2" event={"ID":"7099c17b-763b-4c64-b8af-3c64de776e64","Type":"ContainerStarted","Data":"1466d95e0965488a6d754e2aa0590e690ccc8f1131202edbeee33b736b55e0cc"}
Sep 16 18:59:43.674750 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:43.674729    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 18:59:43.816766 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:43.816694    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2" podStartSLOduration=1.816671967 podStartE2EDuration="1.816671967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:43.813699441 +0000 UTC m=+1216.181755343" watchObservedRunningTime="2026-09-16 18:59:43.816671967 +0000 UTC m=+1216.184727872"
Sep 16 18:59:43.856832 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:43.856801    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:45.418315 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:45.418278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:45.655436 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:45.655397    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:45.791550 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:45.791519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:47.124873 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:47.124836    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:48.357215 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:48.357178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:49.927700 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:49.927665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:51.020949 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:51.020918    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:51.173269 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:51.173234    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:51.485092 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:51.485041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:51.903996 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:51.903883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:52.040483 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.040455    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:52.223913 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.223883    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"]
Sep 16 18:59:52.227359 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.227329    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:52.230665 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.230646    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8bed97ae-dockercfg-lqgjt\""
Sep 16 18:59:52.230756 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.230671    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8bed97ae\""
Sep 16 18:59:52.246378 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.246354    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"]
Sep 16 18:59:52.305316 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.305287    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a166bc-4e23-4153-84c2-aca40641600b-plugins-config-volume\") pod \"payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp\" (UID: \"23a166bc-4e23-4153-84c2-aca40641600b\") " pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:52.305439 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.305333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg6d5\" (UniqueName: \"kubernetes.io/projected/23a166bc-4e23-4153-84c2-aca40641600b-kube-api-access-lg6d5\") pod \"payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp\" (UID: \"23a166bc-4e23-4153-84c2-aca40641600b\") " pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:52.406294 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.406258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg6d5\" (UniqueName: \"kubernetes.io/projected/23a166bc-4e23-4153-84c2-aca40641600b-kube-api-access-lg6d5\") pod \"payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp\" (UID: \"23a166bc-4e23-4153-84c2-aca40641600b\") " pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:52.406431 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.406350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a166bc-4e23-4153-84c2-aca40641600b-plugins-config-volume\") pod \"payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp\" (UID: \"23a166bc-4e23-4153-84c2-aca40641600b\") " pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:52.406950 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.406932    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a166bc-4e23-4153-84c2-aca40641600b-plugins-config-volume\") pod \"payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp\" (UID: \"23a166bc-4e23-4153-84c2-aca40641600b\") " pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:52.419862 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.419838    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg6d5\" (UniqueName: \"kubernetes.io/projected/23a166bc-4e23-4153-84c2-aca40641600b-kube-api-access-lg6d5\") pod \"payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp\" (UID: \"23a166bc-4e23-4153-84c2-aca40641600b\") " pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:52.521765 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.521664    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:52.521765 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.521720    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:52.534141 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.534114    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:52.538301 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.538285    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:52.697888 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.697851    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:52.722237 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.722208    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 18:59:52.827515 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:52.827341    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"]
Sep 16 18:59:52.830250 ip-10-0-141-134 kubenswrapper[2559]: W0916 18:59:52.830216    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a166bc_4e23_4153_84c2_aca40641600b.slice/crio-ec82093707aae32b8ac90379b547bd4fd6af2277993eb3fc58c18c11f25cbb75 WatchSource:0}: Error finding container ec82093707aae32b8ac90379b547bd4fd6af2277993eb3fc58c18c11f25cbb75: Status 404 returned error can't find the container with id ec82093707aae32b8ac90379b547bd4fd6af2277993eb3fc58c18c11f25cbb75
Sep 16 18:59:53.717873 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:53.717831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp" event={"ID":"23a166bc-4e23-4153-84c2-aca40641600b","Type":"ContainerStarted","Data":"7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2"}
Sep 16 18:59:53.718343 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:53.717903    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 18:59:53.718343 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:53.717929    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp" event={"ID":"23a166bc-4e23-4153-84c2-aca40641600b","Type":"ContainerStarted","Data":"ec82093707aae32b8ac90379b547bd4fd6af2277993eb3fc58c18c11f25cbb75"}
Sep 16 18:59:54.246601 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:54.246551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:55.374354 ip-10-0-141-134 kubenswrapper[2559]: I0916 18:59:55.374317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:00.187892 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.187833    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp" podStartSLOduration=8.187814233 podStartE2EDuration="8.187814233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:53.778071231 +0000 UTC m=+1226.146127152" watchObservedRunningTime="2026-09-16 19:00:00.187814233 +0000 UTC m=+1232.555870134"
Sep 16 19:00:00.190285 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.188537    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"]
Sep 16 19:00:00.191671 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.191654    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"
Sep 16 19:00:00.195178 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.195149    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:00:00.201309 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.201292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"
Sep 16 19:00:00.210824 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.210797    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"]
Sep 16 19:00:00.215615 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.215587    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"]
Sep 16 19:00:00.215738 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.215700    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"
Sep 16 19:00:00.229997 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.229946    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"]
Sep 16 19:00:00.240968 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.240942    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"
Sep 16 19:00:00.256362 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.256326    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"]
Sep 16 19:00:00.263140 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.263106    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"
Sep 16 19:00:00.273597 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.273519    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"]
Sep 16 19:00:00.285213 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.284965    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"
Sep 16 19:00:00.490432 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.490405    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"]
Sep 16 19:00:00.493417 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:00:00.493390    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcfbf7f_4c20_44be_98d2_f26a184b28fb.slice/crio-e1c6ed07c266f0cf391157a72377816aeb24ccc985734f6964ad8def633df1f1 WatchSource:0}: Error finding container e1c6ed07c266f0cf391157a72377816aeb24ccc985734f6964ad8def633df1f1: Status 404 returned error can't find the container with id e1c6ed07c266f0cf391157a72377816aeb24ccc985734f6964ad8def633df1f1
Sep 16 19:00:00.535531 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.535502    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"]
Sep 16 19:00:00.537324 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:00:00.537298    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a41f00_6bcf_4445_864c_d5f147f682f4.slice/crio-56570944220adfefbbf01aabce42d1facee55b6b62521ff952da5d2e4f409b2e WatchSource:0}: Error finding container 56570944220adfefbbf01aabce42d1facee55b6b62521ff952da5d2e4f409b2e: Status 404 returned error can't find the container with id 56570944220adfefbbf01aabce42d1facee55b6b62521ff952da5d2e4f409b2e
Sep 16 19:00:00.592020 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.591962    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"]
Sep 16 19:00:00.594159 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:00:00.594135    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3da3c8_e3d5_4873_80f6_e2947b9c8a3d.slice/crio-5a2a421f261486df4a731ae5b4dda28dbf813c7531b82d0727e75cc572e77360 WatchSource:0}: Error finding container 5a2a421f261486df4a731ae5b4dda28dbf813c7531b82d0727e75cc572e77360: Status 404 returned error can't find the container with id 5a2a421f261486df4a731ae5b4dda28dbf813c7531b82d0727e75cc572e77360
Sep 16 19:00:00.752769 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.752671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd" event={"ID":"1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d","Type":"ContainerStarted","Data":"5a2a421f261486df4a731ae5b4dda28dbf813c7531b82d0727e75cc572e77360"}
Sep 16 19:00:00.753944 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.753922    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht" event={"ID":"04a41f00-6bcf-4445-864c-d5f147f682f4","Type":"ContainerStarted","Data":"56570944220adfefbbf01aabce42d1facee55b6b62521ff952da5d2e4f409b2e"}
Sep 16 19:00:00.755031 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:00.755008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk" event={"ID":"5dcfbf7f-4c20-44be-98d2-f26a184b28fb","Type":"ContainerStarted","Data":"e1c6ed07c266f0cf391157a72377816aeb24ccc985734f6964ad8def633df1f1"}
Sep 16 19:00:03.775007 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:03.774944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd" event={"ID":"1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d","Type":"ContainerStarted","Data":"a13a911032e58ea90e4e2418911326206e4b0246503aa1965701061c8f1aaef2"}
Sep 16 19:00:03.776548 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:03.776522    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht" event={"ID":"04a41f00-6bcf-4445-864c-d5f147f682f4","Type":"ContainerStarted","Data":"ebbfdf86b74f2c7ae1a185fecacf3eba145436141b06c9ef1db5ba851f978a48"}
Sep 16 19:00:03.778070 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:03.778048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk" event={"ID":"5dcfbf7f-4c20-44be-98d2-f26a184b28fb","Type":"ContainerStarted","Data":"1b56bd4e3e210fbd68e55d2e2f0abc65bbe0bf38cfc8a84a96cb53e8e4ae1ac0"}
Sep 16 19:00:03.796409 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:03.796366    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd" podStartSLOduration=1.088186216 podStartE2EDuration="3.796352112s" totalImagesPullingTime="2.708165896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:03.794584048 +0000 UTC m=+1236.162639950" watchObservedRunningTime="2026-09-16 19:00:03.796352112 +0000 UTC m=+1236.164408007"
Sep 16 19:00:03.822957 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:03.822899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk" podStartSLOduration=1.136820073 podStartE2EDuration="3.822883648s" totalImagesPullingTime="2.686063575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:03.81751339 +0000 UTC m=+1236.185569292" watchObservedRunningTime="2026-09-16 19:00:03.822883648 +0000 UTC m=+1236.190939549"
Sep 16 19:00:03.840215 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:03.840155    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht" podStartSLOduration=1.203629893 podStartE2EDuration="3.840134808s" totalImagesPullingTime="2.636504915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:03.837526952 +0000 UTC m=+1236.205582853" watchObservedRunningTime="2026-09-16 19:00:03.840134808 +0000 UTC m=+1236.208190710"
Sep 16 19:00:04.725639 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:04.725609    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 19:00:04.784949 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:04.784916    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d" containerID="a13a911032e58ea90e4e2418911326206e4b0246503aa1965701061c8f1aaef2" exitCode=0
Sep 16 19:00:04.785347 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:04.785013    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd" event={"ID":"1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d","Type":"ContainerDied","Data":"a13a911032e58ea90e4e2418911326206e4b0246503aa1965701061c8f1aaef2"}
Sep 16 19:00:04.786612 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:04.786584    2559 generic.go:356] "Generic (PLEG): container finished" podID="04a41f00-6bcf-4445-864c-d5f147f682f4" containerID="ebbfdf86b74f2c7ae1a185fecacf3eba145436141b06c9ef1db5ba851f978a48" exitCode=0
Sep 16 19:00:04.786737 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:04.786662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht" event={"ID":"04a41f00-6bcf-4445-864c-d5f147f682f4","Type":"ContainerDied","Data":"ebbfdf86b74f2c7ae1a185fecacf3eba145436141b06c9ef1db5ba851f978a48"}
Sep 16 19:00:04.788275 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:04.788244    2559 generic.go:356] "Generic (PLEG): container finished" podID="5dcfbf7f-4c20-44be-98d2-f26a184b28fb" containerID="1b56bd4e3e210fbd68e55d2e2f0abc65bbe0bf38cfc8a84a96cb53e8e4ae1ac0" exitCode=0
Sep 16 19:00:04.788386 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:04.788283    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk" event={"ID":"5dcfbf7f-4c20-44be-98d2-f26a184b28fb","Type":"ContainerDied","Data":"1b56bd4e3e210fbd68e55d2e2f0abc65bbe0bf38cfc8a84a96cb53e8e4ae1ac0"}
Sep 16 19:00:05.958916 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:05.958893    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"
Sep 16 19:00:05.962409 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:05.962388    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"
Sep 16 19:00:05.992558 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:05.992532    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"
Sep 16 19:00:06.798747 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.798719    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"
Sep 16 19:00:06.798945 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.798714    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd" event={"ID":"1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d","Type":"ContainerDied","Data":"5a2a421f261486df4a731ae5b4dda28dbf813c7531b82d0727e75cc572e77360"}
Sep 16 19:00:06.798945 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.798858    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a2a421f261486df4a731ae5b4dda28dbf813c7531b82d0727e75cc572e77360"
Sep 16 19:00:06.800072 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.800055    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"
Sep 16 19:00:06.800207 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.800099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht" event={"ID":"04a41f00-6bcf-4445-864c-d5f147f682f4","Type":"ContainerDied","Data":"56570944220adfefbbf01aabce42d1facee55b6b62521ff952da5d2e4f409b2e"}
Sep 16 19:00:06.800207 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.800151    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56570944220adfefbbf01aabce42d1facee55b6b62521ff952da5d2e4f409b2e"
Sep 16 19:00:06.801491 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.801474    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"
Sep 16 19:00:06.801572 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.801496    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk" event={"ID":"5dcfbf7f-4c20-44be-98d2-f26a184b28fb","Type":"ContainerDied","Data":"e1c6ed07c266f0cf391157a72377816aeb24ccc985734f6964ad8def633df1f1"}
Sep 16 19:00:06.801572 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:06.801538    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1c6ed07c266f0cf391157a72377816aeb24ccc985734f6964ad8def633df1f1"
Sep 16 19:00:13.096870 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.096836    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"]
Sep 16 19:00:13.097264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097190    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d" containerName="cleanup"
Sep 16 19:00:13.097264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097202    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d" containerName="cleanup"
Sep 16 19:00:13.097264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097214    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcfbf7f-4c20-44be-98d2-f26a184b28fb" containerName="cleanup"
Sep 16 19:00:13.097264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097220    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcfbf7f-4c20-44be-98d2-f26a184b28fb" containerName="cleanup"
Sep 16 19:00:13.097264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097230    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a41f00-6bcf-4445-864c-d5f147f682f4" containerName="cleanup"
Sep 16 19:00:13.097264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097236    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a41f00-6bcf-4445-864c-d5f147f682f4" containerName="cleanup"
Sep 16 19:00:13.097444 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097307    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d" containerName="cleanup"
Sep 16 19:00:13.097444 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097317    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a41f00-6bcf-4445-864c-d5f147f682f4" containerName="cleanup"
Sep 16 19:00:13.097444 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.097324    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcfbf7f-4c20-44be-98d2-f26a184b28fb" containerName="cleanup"
Sep 16 19:00:13.100346 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.100330    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.103514 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.103483    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2efa6f\"/\"default-dockercfg-ctbbn\""
Sep 16 19:00:13.103627 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.103563    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-2efa6f\"/\"autores-model-2efa6f-kserve-self-signed-certs\""
Sep 16 19:00:13.104784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.104763    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2efa6f\"/\"kube-root-ca.crt\""
Sep 16 19:00:13.104894 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.104844    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-2efa6f\"/\"openshift-service-ca.crt\""
Sep 16 19:00:13.122418 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.122395    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"]
Sep 16 19:00:13.190188 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.190158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trn7\" (UniqueName: \"kubernetes.io/projected/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-kube-api-access-7trn7\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.190374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.190199    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-dshm\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.190374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.190286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tmp-dir\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.190374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.190336    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-model-cache\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.190502 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.190404    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-home\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.190502 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.190437    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tls-certs\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.289456 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.289426    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:13.291534 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.291514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-home\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.291584 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.291545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tls-certs\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.291584 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.291567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trn7\" (UniqueName: \"kubernetes.io/projected/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-kube-api-access-7trn7\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.291695 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.291592    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-dshm\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.291695 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.291641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tmp-dir\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.291695 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.291684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-model-cache\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.291961 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.291940    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-home\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.292086 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.292066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tmp-dir\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.292125 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.292082    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-model-cache\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.293888 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.293868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-dshm\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.294069 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.294053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tls-certs\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.307016 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.306983    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trn7\" (UniqueName: \"kubernetes.io/projected/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-kube-api-access-7trn7\") pod \"autores-model-2efa6f-kserve-6dd547f555-8mnmc\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") " pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.409216 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.409141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:13.691003 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.690966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"]
Sep 16 19:00:13.692548 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:00:13.692524    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d9da3e7_16f0_4d1c_94ca_259d2b67366e.slice/crio-1237c5b5b7832d42909b7b4ada2c7b49d3373715d83320f8e785fded57f4a226 WatchSource:0}: Error finding container 1237c5b5b7832d42909b7b4ada2c7b49d3373715d83320f8e785fded57f4a226: Status 404 returned error can't find the container with id 1237c5b5b7832d42909b7b4ada2c7b49d3373715d83320f8e785fded57f4a226
Sep 16 19:00:13.836440 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.836403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc" event={"ID":"7d9da3e7-16f0-4d1c-94ca-259d2b67366e","Type":"ContainerStarted","Data":"353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda"}
Sep 16 19:00:13.836604 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.836457    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc" event={"ID":"7d9da3e7-16f0-4d1c-94ca-259d2b67366e","Type":"ContainerStarted","Data":"1237c5b5b7832d42909b7b4ada2c7b49d3373715d83320f8e785fded57f4a226"}
Sep 16 19:00:13.871504 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:13.871447    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc" podStartSLOduration=0.87143168 podStartE2EDuration="871.43168ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:13.869805506 +0000 UTC m=+1246.237861398" watchObservedRunningTime="2026-09-16 19:00:13.87143168 +0000 UTC m=+1246.239487581"
Sep 16 19:00:14.565587 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:14.565552    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:16.122064 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:16.122030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:17.325282 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:17.325244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:18.527464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:18.527432    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:19.386608 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:19.386574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:20.458656 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:20.458598    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:21.493086 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:21.493050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:22.512158 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:22.512124    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:23.114576 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:23.114541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:23.410297 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:23.410207    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:23.410297 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:23.410251    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:23.422799 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:23.422772    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:23.895630 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:23.895602    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:00:24.163428 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:24.163334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:27.226438 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:27.226405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:28.044876 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.044845    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:28.143602 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.143566    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"]
Sep 16 19:00:28.143822 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.143801    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp" podUID="23a166bc-4e23-4153-84c2-aca40641600b" containerName="payload-processing" containerID="cri-o://7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2" gracePeriod=30
Sep 16 19:00:28.196348 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.196316    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:28.309639 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.309556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:28.392871 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.392848    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 19:00:28.526934 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.526892    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg6d5\" (UniqueName: \"kubernetes.io/projected/23a166bc-4e23-4153-84c2-aca40641600b-kube-api-access-lg6d5\") pod \"23a166bc-4e23-4153-84c2-aca40641600b\" (UID: \"23a166bc-4e23-4153-84c2-aca40641600b\") "
Sep 16 19:00:28.527165 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.526947    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a166bc-4e23-4153-84c2-aca40641600b-plugins-config-volume\") pod \"23a166bc-4e23-4153-84c2-aca40641600b\" (UID: \"23a166bc-4e23-4153-84c2-aca40641600b\") "
Sep 16 19:00:28.527371 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.527344    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a166bc-4e23-4153-84c2-aca40641600b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23a166bc-4e23-4153-84c2-aca40641600b" (UID: "23a166bc-4e23-4153-84c2-aca40641600b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:28.529059 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.529034    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a166bc-4e23-4153-84c2-aca40641600b-kube-api-access-lg6d5" (OuterVolumeSpecName: "kube-api-access-lg6d5") pod "23a166bc-4e23-4153-84c2-aca40641600b" (UID: "23a166bc-4e23-4153-84c2-aca40641600b"). InnerVolumeSpecName "kube-api-access-lg6d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:28.628171 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.628076    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg6d5\" (UniqueName: \"kubernetes.io/projected/23a166bc-4e23-4153-84c2-aca40641600b-kube-api-access-lg6d5\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.628171 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.628111    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23a166bc-4e23-4153-84c2-aca40641600b-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:00:28.909206 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.909118    2559 generic.go:356] "Generic (PLEG): container finished" podID="23a166bc-4e23-4153-84c2-aca40641600b" containerID="7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2" exitCode=0
Sep 16 19:00:28.909206 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.909161    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"
Sep 16 19:00:28.909390 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.909205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp" event={"ID":"23a166bc-4e23-4153-84c2-aca40641600b","Type":"ContainerDied","Data":"7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2"}
Sep 16 19:00:28.909390 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.909260    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp" event={"ID":"23a166bc-4e23-4153-84c2-aca40641600b","Type":"ContainerDied","Data":"ec82093707aae32b8ac90379b547bd4fd6af2277993eb3fc58c18c11f25cbb75"}
Sep 16 19:00:28.909390 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.909285    2559 scope.go:117] "RemoveContainer" containerID="7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2"
Sep 16 19:00:28.918877 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.918846    2559 scope.go:117] "RemoveContainer" containerID="7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2"
Sep 16 19:00:28.919220 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:00:28.919195    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2\": container with ID starting with 7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2 not found: ID does not exist" containerID="7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2"
Sep 16 19:00:28.919296 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.919228    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2"} err="failed to get container status \"7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2\": rpc error: code = NotFound desc = could not find container \"7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2\": container with ID starting with 7be8b3bf8181d74a7826e1ba7694dc3d017845540bd25f90ce967d6462cdbfd2 not found: ID does not exist"
Sep 16 19:00:28.943110 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.943087    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"]
Sep 16 19:00:28.951515 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:28.951494    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8bed97ae-6c94f9f5d9-m28dp"]
Sep 16 19:00:29.040519 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:29.040487    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:29.210925 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:29.210896    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:29.327653 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:29.327613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.224776 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:30.224740    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a166bc-4e23-4153-84c2-aca40641600b" path="/var/lib/kubelet/pods/23a166bc-4e23-4153-84c2-aca40641600b/volumes"
Sep 16 19:00:30.420343 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:30.420312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.605150 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:30.605077    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.729203 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:30.729164    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:30.908628 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:30.908555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:31.133667 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:31.133634    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:32.699535 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:32.699507    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:33.863667 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:33.863630    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:34.013647 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:34.013616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:35.868552 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:35.868518    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:37.411574 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:37.411538    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:39.024706 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:39.024676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:39.175832 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:39.175790    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:40.089112 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:40.089076    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:40.271260 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:40.271228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:41.184621 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:41.184590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:42.732358 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:42.732330    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:43.847161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:43.847130    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:45.372676 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:45.372639    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:46.726089 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:46.726056    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:48.233082 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:48.233046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:49.569368 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:49.569335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:51.330291 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:51.330254    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:52.299117 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:52.299080    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:53.914875 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:53.914845    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:54.915866 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:54.915832    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:55.769467 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:55.769432    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"]
Sep 16 19:00:55.769715 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:55.769685    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k" podUID="d0478f39-b092-49ad-8576-f581937b3669" containerName="payload-processing" containerID="cri-o://ae2ab5c05456de5e2bc7a915d30278684aab49e79230d7e7dd8c77aff47c6fb3" gracePeriod=30
Sep 16 19:00:55.783349 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:55.783320    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"]
Sep 16 19:00:55.783570 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:55.783545    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk" podUID="2c92eb9b-2a5b-4ae8-843b-abbebf1cc146" containerName="payload-pre-processing" containerID="cri-o://1d6a05d37804be038892ae4a1083b3903ba5f7181871b1176004688a1b2d02df" gracePeriod=30
Sep 16 19:00:55.878899 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:55.878865    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:56.023224 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.023149    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:56.040047 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.040019    2559 generic.go:356] "Generic (PLEG): container finished" podID="d0478f39-b092-49ad-8576-f581937b3669" containerID="ae2ab5c05456de5e2bc7a915d30278684aab49e79230d7e7dd8c77aff47c6fb3" exitCode=0
Sep 16 19:00:56.040171 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.040090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k" event={"ID":"d0478f39-b092-49ad-8576-f581937b3669","Type":"ContainerDied","Data":"ae2ab5c05456de5e2bc7a915d30278684aab49e79230d7e7dd8c77aff47c6fb3"}
Sep 16 19:00:56.040171 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.040159    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k" event={"ID":"d0478f39-b092-49ad-8576-f581937b3669","Type":"ContainerDied","Data":"65b4efdd633ea59e91511fe5e9ccb8caf92a5d5322875203027ba3a79b56d142"}
Sep 16 19:00:56.040279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.040179    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b4efdd633ea59e91511fe5e9ccb8caf92a5d5322875203027ba3a79b56d142"
Sep 16 19:00:56.041790 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.041769    2559 generic.go:356] "Generic (PLEG): container finished" podID="2c92eb9b-2a5b-4ae8-843b-abbebf1cc146" containerID="1d6a05d37804be038892ae4a1083b3903ba5f7181871b1176004688a1b2d02df" exitCode=0
Sep 16 19:00:56.041897 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.041846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk" event={"ID":"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146","Type":"ContainerDied","Data":"1d6a05d37804be038892ae4a1083b3903ba5f7181871b1176004688a1b2d02df"}
Sep 16 19:00:56.056290 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.056270    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 19:00:56.059113 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.059094    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 19:00:56.154157 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.154124    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0478f39-b092-49ad-8576-f581937b3669-plugins-config-volume\") pod \"d0478f39-b092-49ad-8576-f581937b3669\" (UID: \"d0478f39-b092-49ad-8576-f581937b3669\") "
Sep 16 19:00:56.154356 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.154163    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vcjw\" (UniqueName: \"kubernetes.io/projected/d0478f39-b092-49ad-8576-f581937b3669-kube-api-access-2vcjw\") pod \"d0478f39-b092-49ad-8576-f581937b3669\" (UID: \"d0478f39-b092-49ad-8576-f581937b3669\") "
Sep 16 19:00:56.154356 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.154194    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-plugins-config-volume\") pod \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\" (UID: \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\") "
Sep 16 19:00:56.154356 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.154271    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vn7tt\" (UniqueName: \"kubernetes.io/projected/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-kube-api-access-vn7tt\") pod \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\" (UID: \"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146\") "
Sep 16 19:00:56.154615 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.154585    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0478f39-b092-49ad-8576-f581937b3669-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0478f39-b092-49ad-8576-f581937b3669" (UID: "d0478f39-b092-49ad-8576-f581937b3669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:56.154673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.154628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c92eb9b-2a5b-4ae8-843b-abbebf1cc146" (UID: "2c92eb9b-2a5b-4ae8-843b-abbebf1cc146"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:56.156422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.156397    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-kube-api-access-vn7tt" (OuterVolumeSpecName: "kube-api-access-vn7tt") pod "2c92eb9b-2a5b-4ae8-843b-abbebf1cc146" (UID: "2c92eb9b-2a5b-4ae8-843b-abbebf1cc146"). InnerVolumeSpecName "kube-api-access-vn7tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:56.156519 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.156406    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0478f39-b092-49ad-8576-f581937b3669-kube-api-access-2vcjw" (OuterVolumeSpecName: "kube-api-access-2vcjw") pod "d0478f39-b092-49ad-8576-f581937b3669" (UID: "d0478f39-b092-49ad-8576-f581937b3669"). InnerVolumeSpecName "kube-api-access-2vcjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:56.255691 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.255654    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7tt\" (UniqueName: \"kubernetes.io/projected/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-kube-api-access-vn7tt\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:00:56.255691 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.255685    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0478f39-b092-49ad-8576-f581937b3669-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:00:56.255691 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.255695    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vcjw\" (UniqueName: \"kubernetes.io/projected/d0478f39-b092-49ad-8576-f581937b3669-kube-api-access-2vcjw\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:00:56.255917 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.255703    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:00:56.328319 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.328230    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:56.596125 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.596034    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:56.924090 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:56.923956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:57.048888 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:57.048852    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"
Sep 16 19:00:57.048888 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:57.048878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk" event={"ID":"2c92eb9b-2a5b-4ae8-843b-abbebf1cc146","Type":"ContainerDied","Data":"a23907c73d8acfa3f1ac49d8a110af73f28a6a34746a31b26a8fb00adbc65ea3"}
Sep 16 19:00:57.049439 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:57.048991    2559 scope.go:117] "RemoveContainer" containerID="1d6a05d37804be038892ae4a1083b3903ba5f7181871b1176004688a1b2d02df"
Sep 16 19:00:57.049439 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:57.049063    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"
Sep 16 19:00:57.067169 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:57.067144    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"]
Sep 16 19:00:57.072154 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:57.072132    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-2efa6f-76f4bfc89-jcw2k"]
Sep 16 19:00:57.089842 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:57.089821    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"]
Sep 16 19:00:57.095051 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:57.095032    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-2efa6f-8576698bf4-rnkjk"]
Sep 16 19:00:58.040624 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.040594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:58.222579 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.222544    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c92eb9b-2a5b-4ae8-843b-abbebf1cc146" path="/var/lib/kubelet/pods/2c92eb9b-2a5b-4ae8-843b-abbebf1cc146/volumes"
Sep 16 19:00:58.222921 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.222872    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0478f39-b092-49ad-8576-f581937b3669" path="/var/lib/kubelet/pods/d0478f39-b092-49ad-8576-f581937b3669/volumes"
Sep 16 19:00:58.813728 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.813695    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"]
Sep 16 19:00:58.814052 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814039    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c92eb9b-2a5b-4ae8-843b-abbebf1cc146" containerName="payload-pre-processing"
Sep 16 19:00:58.814117 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814054    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c92eb9b-2a5b-4ae8-843b-abbebf1cc146" containerName="payload-pre-processing"
Sep 16 19:00:58.814117 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814064    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a166bc-4e23-4153-84c2-aca40641600b" containerName="payload-processing"
Sep 16 19:00:58.814117 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814069    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a166bc-4e23-4153-84c2-aca40641600b" containerName="payload-processing"
Sep 16 19:00:58.814117 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814084    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0478f39-b092-49ad-8576-f581937b3669" containerName="payload-processing"
Sep 16 19:00:58.814117 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814090    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0478f39-b092-49ad-8576-f581937b3669" containerName="payload-processing"
Sep 16 19:00:58.814320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814155    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0478f39-b092-49ad-8576-f581937b3669" containerName="payload-processing"
Sep 16 19:00:58.814320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814165    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c92eb9b-2a5b-4ae8-843b-abbebf1cc146" containerName="payload-pre-processing"
Sep 16 19:00:58.814320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.814171    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a166bc-4e23-4153-84c2-aca40641600b" containerName="payload-processing"
Sep 16 19:00:58.817219 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.817203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"
Sep 16 19:00:58.820682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.820660    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:00:58.820982 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.820949    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:00:58.828337 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.828313    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"]
Sep 16 19:00:58.881114 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.881080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3654ee0-7298-440f-a124-9459a5ed2679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-96b42948-jnw6s\" (UID: \"c3654ee0-7298-440f-a124-9459a5ed2679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"
Sep 16 19:00:58.982476 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.982435    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3654ee0-7298-440f-a124-9459a5ed2679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-96b42948-jnw6s\" (UID: \"c3654ee0-7298-440f-a124-9459a5ed2679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"
Sep 16 19:00:58.983131 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:58.983107    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3654ee0-7298-440f-a124-9459a5ed2679-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-96b42948-jnw6s\" (UID: \"c3654ee0-7298-440f-a124-9459a5ed2679\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"
Sep 16 19:00:59.127877 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:59.127786    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"
Sep 16 19:00:59.408580 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:59.408554    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"]
Sep 16 19:00:59.411046 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:00:59.411020    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3654ee0_7298_440f_a124_9459a5ed2679.slice/crio-1a44f9f3969bc0bec759c47d98b3d16c84fd7a92fb9930dd3432261ed8180df6 WatchSource:0}: Error finding container 1a44f9f3969bc0bec759c47d98b3d16c84fd7a92fb9930dd3432261ed8180df6: Status 404 returned error can't find the container with id 1a44f9f3969bc0bec759c47d98b3d16c84fd7a92fb9930dd3432261ed8180df6
Sep 16 19:00:59.535074 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:00:59.535040    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:00.067203 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:00.067175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s" event={"ID":"c3654ee0-7298-440f-a124-9459a5ed2679","Type":"ContainerStarted","Data":"c2eae606a74a4992294938be0528b1f339883d0fd0749a1dd402b5b42b0b520e"}
Sep 16 19:01:00.067306 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:00.067226    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s" event={"ID":"c3654ee0-7298-440f-a124-9459a5ed2679","Type":"ContainerStarted","Data":"1a44f9f3969bc0bec759c47d98b3d16c84fd7a92fb9930dd3432261ed8180df6"}
Sep 16 19:01:00.734354 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:00.734322    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:01.074114 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.074026    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3654ee0-7298-440f-a124-9459a5ed2679" containerID="c2eae606a74a4992294938be0528b1f339883d0fd0749a1dd402b5b42b0b520e" exitCode=0
Sep 16 19:01:01.074114 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.074071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s" event={"ID":"c3654ee0-7298-440f-a124-9459a5ed2679","Type":"ContainerDied","Data":"c2eae606a74a4992294938be0528b1f339883d0fd0749a1dd402b5b42b0b520e"}
Sep 16 19:01:01.198041 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.198010    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"
Sep 16 19:01:01.304864 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.304829    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3654ee0-7298-440f-a124-9459a5ed2679-maas-api-service-ca\") pod \"c3654ee0-7298-440f-a124-9459a5ed2679\" (UID: \"c3654ee0-7298-440f-a124-9459a5ed2679\") "
Sep 16 19:01:01.305249 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.305224    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3654ee0-7298-440f-a124-9459a5ed2679-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3654ee0-7298-440f-a124-9459a5ed2679" (UID: "c3654ee0-7298-440f-a124-9459a5ed2679"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:01.406493 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.406404    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3654ee0-7298-440f-a124-9459a5ed2679-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:01.583914 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.583876    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"]
Sep 16 19:01:01.723508 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.723473    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:01.821274 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:01.821236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:02.080040 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:02.079957    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"
Sep 16 19:01:02.080040 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:02.079965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s" event={"ID":"c3654ee0-7298-440f-a124-9459a5ed2679","Type":"ContainerDied","Data":"1a44f9f3969bc0bec759c47d98b3d16c84fd7a92fb9930dd3432261ed8180df6"}
Sep 16 19:01:02.080040 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:02.080026    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a44f9f3969bc0bec759c47d98b3d16c84fd7a92fb9930dd3432261ed8180df6"
Sep 16 19:01:03.067594 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:03.067523    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:04.089340 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:04.089301    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:05.257243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:05.257208    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:06.310625 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:06.310583    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:06.490590 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:06.490548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:06.599964 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:06.599879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:06.728582 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:06.728541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:06.780153 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:06.780113    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"]
Sep 16 19:01:06.786113 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:06.786087    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-96b42948-jnw6s"]
Sep 16 19:01:06.848281 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:06.848238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:07.012230 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:07.012192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:07.831727 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:07.831684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:08.005241 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:08.005205    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:08.222436 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:08.222405    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3654ee0-7298-440f-a124-9459a5ed2679" path="/var/lib/kubelet/pods/c3654ee0-7298-440f-a124-9459a5ed2679/volumes"
Sep 16 19:01:08.649415 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:08.649333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:08.805758 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:08.805725    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:09.555289 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:09.555244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:09.910916 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:09.910818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:10.958623 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:10.958592    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:12.381359 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:12.381324    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:12.483377 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:12.483335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:16.602571 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.602533    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc" podUID="7d9da3e7-16f0-4d1c-94ca-259d2b67366e" containerName="main" containerID="cri-o://353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda" gracePeriod=45
Sep 16 19:01:16.635526 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.635498    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:16.762288 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.762252    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:16.858617 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.858550    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:01:16.893541 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.893515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:16.933902 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.933862    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-model-cache\") pod \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") "
Sep 16 19:01:16.934084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.933908    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-home\") pod \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") "
Sep 16 19:01:16.934084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.933949    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tls-certs\") pod \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") "
Sep 16 19:01:16.934084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.934035    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-dshm\") pod \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") "
Sep 16 19:01:16.934221 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.934085    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d9da3e7-16f0-4d1c-94ca-259d2b67366e" (UID: "7d9da3e7-16f0-4d1c-94ca-259d2b67366e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:16.934221 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.934094    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tmp-dir\") pod \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") "
Sep 16 19:01:16.934221 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.934150    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7trn7\" (UniqueName: \"kubernetes.io/projected/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-kube-api-access-7trn7\") pod \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\" (UID: \"7d9da3e7-16f0-4d1c-94ca-259d2b67366e\") "
Sep 16 19:01:16.934221 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.934207    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-home" (OuterVolumeSpecName: "home") pod "7d9da3e7-16f0-4d1c-94ca-259d2b67366e" (UID: "7d9da3e7-16f0-4d1c-94ca-259d2b67366e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:16.934461 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.934392    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:16.934461 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.934414    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:16.934461 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.934418    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d9da3e7-16f0-4d1c-94ca-259d2b67366e" (UID: "7d9da3e7-16f0-4d1c-94ca-259d2b67366e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:16.936365 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.936345    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d9da3e7-16f0-4d1c-94ca-259d2b67366e" (UID: "7d9da3e7-16f0-4d1c-94ca-259d2b67366e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:16.936365 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.936354    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-dshm" (OuterVolumeSpecName: "dshm") pod "7d9da3e7-16f0-4d1c-94ca-259d2b67366e" (UID: "7d9da3e7-16f0-4d1c-94ca-259d2b67366e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:16.936474 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.936379    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-kube-api-access-7trn7" (OuterVolumeSpecName: "kube-api-access-7trn7") pod "7d9da3e7-16f0-4d1c-94ca-259d2b67366e" (UID: "7d9da3e7-16f0-4d1c-94ca-259d2b67366e"). InnerVolumeSpecName "kube-api-access-7trn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:16.997722 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:16.997687    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:17.035873 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.035838    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:17.035873 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.035869    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:17.036051 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.035881    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trn7\" (UniqueName: \"kubernetes.io/projected/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-kube-api-access-7trn7\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:17.036051 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.035889    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9da3e7-16f0-4d1c-94ca-259d2b67366e-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:01:17.156799 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.156716    2559 generic.go:356] "Generic (PLEG): container finished" podID="7d9da3e7-16f0-4d1c-94ca-259d2b67366e" containerID="353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda" exitCode=0
Sep 16 19:01:17.156799 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.156763    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"
Sep 16 19:01:17.157004 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.156806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc" event={"ID":"7d9da3e7-16f0-4d1c-94ca-259d2b67366e","Type":"ContainerDied","Data":"353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda"}
Sep 16 19:01:17.157004 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.156854    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc" event={"ID":"7d9da3e7-16f0-4d1c-94ca-259d2b67366e","Type":"ContainerDied","Data":"1237c5b5b7832d42909b7b4ada2c7b49d3373715d83320f8e785fded57f4a226"}
Sep 16 19:01:17.157004 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.156876    2559 scope.go:117] "RemoveContainer" containerID="353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda"
Sep 16 19:01:17.165852 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.165831    2559 scope.go:117] "RemoveContainer" containerID="353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda"
Sep 16 19:01:17.166257 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:01:17.166234    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda\": container with ID starting with 353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda not found: ID does not exist" containerID="353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda"
Sep 16 19:01:17.166331 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.166269    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda"} err="failed to get container status \"353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda\": rpc error: code = NotFound desc = could not find container \"353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda\": container with ID starting with 353e8fe645a69965eb7a598ec9a485006069e1fccf1daf46944ae71413073bda not found: ID does not exist"
Sep 16 19:01:17.182202 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.182177    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"]
Sep 16 19:01:17.186589 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.186566    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-2efa6f/autores-model-2efa6f-kserve-6dd547f555-8mnmc"]
Sep 16 19:01:17.432555 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:17.432467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:18.222121 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:18.222091    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d9da3e7-16f0-4d1c-94ca-259d2b67366e" path="/var/lib/kubelet/pods/7d9da3e7-16f0-4d1c-94ca-259d2b67366e/volumes"
Sep 16 19:01:26.821649 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:26.821616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:26.898819 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:26.898765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:27.929491 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.929449    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"]
Sep 16 19:01:27.930003 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.929957    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9da3e7-16f0-4d1c-94ca-259d2b67366e" containerName="main"
Sep 16 19:01:27.930003 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.929993    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9da3e7-16f0-4d1c-94ca-259d2b67366e" containerName="main"
Sep 16 19:01:27.930156 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.930010    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3654ee0-7298-440f-a124-9459a5ed2679" containerName="revoke-keys"
Sep 16 19:01:27.930156 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.930018    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3654ee0-7298-440f-a124-9459a5ed2679" containerName="revoke-keys"
Sep 16 19:01:27.930156 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.930106    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3654ee0-7298-440f-a124-9459a5ed2679" containerName="revoke-keys"
Sep 16 19:01:27.930156 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.930121    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9da3e7-16f0-4d1c-94ca-259d2b67366e" containerName="main"
Sep 16 19:01:27.932933 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.932913    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:27.937295 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.937277    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1857c\"/\"premium-w0-b1857c-kserve-self-signed-certs\""
Sep 16 19:01:27.977576 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.977548    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:27.978788 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:27.978768    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"]
Sep 16 19:01:28.036708 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.036675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-dshm\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.036708 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.036713    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-model-cache\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.036926 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.036739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tls-certs\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.036926 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.036821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tmp-dir\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.036926 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.036852    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpftg\" (UniqueName: \"kubernetes.io/projected/f80a2441-8cef-4615-aa2b-ebf6eb34385e-kube-api-access-fpftg\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.036926 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.036916    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-home\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.137985 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.137948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tls-certs\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.138135 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.138028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tmp-dir\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.138135 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.138048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpftg\" (UniqueName: \"kubernetes.io/projected/f80a2441-8cef-4615-aa2b-ebf6eb34385e-kube-api-access-fpftg\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.138135 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.138093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-home\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.138135 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.138134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-dshm\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.138320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.138149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-model-cache\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.138490 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.138463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tmp-dir\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.138563 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.138467    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-model-cache\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.138563 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.138511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-home\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.140335 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.140317    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-dshm\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.141540 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.141524    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1857c\"/\"premium-w0-b1857c-kserve-self-signed-certs\""
Sep 16 19:01:28.149631 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.149609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpftg\" (UniqueName: \"kubernetes.io/projected/f80a2441-8cef-4615-aa2b-ebf6eb34385e-kube-api-access-fpftg\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.150849 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.150828    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tls-certs\") pod \"premium-w0-b1857c-kserve-598c7946cb-jhhjg\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") " pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.241560 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.241533    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:28.534029 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:28.533994    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"]
Sep 16 19:01:28.536297 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:01:28.536270    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf80a2441_8cef_4615_aa2b_ebf6eb34385e.slice/crio-320b04dbafe9daa31872a891a91020fff945b52f92e1ddfba81a4fe8d6cbc113 WatchSource:0}: Error finding container 320b04dbafe9daa31872a891a91020fff945b52f92e1ddfba81a4fe8d6cbc113: Status 404 returned error can't find the container with id 320b04dbafe9daa31872a891a91020fff945b52f92e1ddfba81a4fe8d6cbc113
Sep 16 19:01:29.216271 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:29.216238    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg" event={"ID":"f80a2441-8cef-4615-aa2b-ebf6eb34385e","Type":"ContainerStarted","Data":"6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef"}
Sep 16 19:01:29.216271 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:29.216277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg" event={"ID":"f80a2441-8cef-4615-aa2b-ebf6eb34385e","Type":"ContainerStarted","Data":"320b04dbafe9daa31872a891a91020fff945b52f92e1ddfba81a4fe8d6cbc113"}
Sep 16 19:01:29.240858 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:29.240803    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg" podStartSLOduration=2.2407855899999998 podStartE2EDuration="2.24078559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:29.239568448 +0000 UTC m=+1321.607624382" watchObservedRunningTime="2026-09-16 19:01:29.24078559 +0000 UTC m=+1321.608841491"
Sep 16 19:01:34.323091 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:34.323050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:38.242374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:38.242342    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:38.242756 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:38.242388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:38.254856 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:38.254828    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:38.270136 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:38.270109    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:01:40.206754 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:40.206720    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:40.503947 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:40.503868    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:40.990852 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:40.990825    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:41.420367 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:41.420298    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:41.615992 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:41.615950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:43.386451 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:43.386416    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:44.100160 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:44.100122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:44.903902 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:44.903869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.004616 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:45.004585    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:45.295345 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:45.295317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:46.171851 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:46.171813    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:46.334406 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:46.334362    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:47.357007 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:47.356954    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:48.243564 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:48.243530    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:49.354892 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:49.354860    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:49.488423 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:49.488392    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:49.861989 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:49.861941    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:49.976739 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:49.976706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:49.985407 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:49.985382    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"]
Sep 16 19:01:49.988804 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:49.988787    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:49.993210 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:49.993192    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-9d2450\""
Sep 16 19:01:50.015530 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.015504    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"]
Sep 16 19:01:50.030889 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.030859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ed512e-9a9a-497b-a02c-54e59e47c95f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4\" (UID: \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\") " pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:50.031046 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.030904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpvk\" (UniqueName: \"kubernetes.io/projected/d7ed512e-9a9a-497b-a02c-54e59e47c95f-kube-api-access-xkpvk\") pod \"payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4\" (UID: \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\") " pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:50.106192 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.106140    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:50.131440 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.131369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ed512e-9a9a-497b-a02c-54e59e47c95f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4\" (UID: \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\") " pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:50.131440 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.131437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpvk\" (UniqueName: \"kubernetes.io/projected/d7ed512e-9a9a-497b-a02c-54e59e47c95f-kube-api-access-xkpvk\") pod \"payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4\" (UID: \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\") " pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:50.132053 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.132034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ed512e-9a9a-497b-a02c-54e59e47c95f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4\" (UID: \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\") " pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:50.141012 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.140962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpvk\" (UniqueName: \"kubernetes.io/projected/d7ed512e-9a9a-497b-a02c-54e59e47c95f-kube-api-access-xkpvk\") pod \"payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4\" (UID: \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\") " pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:50.299039 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.299007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:50.593794 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:50.593768    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"]
Sep 16 19:01:50.595529 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:01:50.595499    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ed512e_9a9a_497b_a02c_54e59e47c95f.slice/crio-6f456647b5042548e9711c9613d639d683d3cd10e8ecf46c898820b014e9e287 WatchSource:0}: Error finding container 6f456647b5042548e9711c9613d639d683d3cd10e8ecf46c898820b014e9e287: Status 404 returned error can't find the container with id 6f456647b5042548e9711c9613d639d683d3cd10e8ecf46c898820b014e9e287
Sep 16 19:01:51.321885 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:51.321846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4" event={"ID":"d7ed512e-9a9a-497b-a02c-54e59e47c95f","Type":"ContainerStarted","Data":"e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1"}
Sep 16 19:01:51.322112 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:51.321994    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:01:51.322112 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:51.322038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4" event={"ID":"d7ed512e-9a9a-497b-a02c-54e59e47c95f","Type":"ContainerStarted","Data":"6f456647b5042548e9711c9613d639d683d3cd10e8ecf46c898820b014e9e287"}
Sep 16 19:01:51.352394 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:51.352344    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4" podStartSLOduration=2.352328792 podStartE2EDuration="2.352328792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:51.350602067 +0000 UTC m=+1343.718657997" watchObservedRunningTime="2026-09-16 19:01:51.352328792 +0000 UTC m=+1343.720384693"
Sep 16 19:01:51.479724 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:51.479691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:51.586037 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:01:51.585938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:02.329306 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:02.329273    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:02:05.410922 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:05.410891    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:07.354726 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:07.354693    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:13.537526 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:13.537486    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:14.426215 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:14.426185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:14.629481 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:14.629449    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:14.733723 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:14.733688    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:14.849174 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:14.849137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:15.002305 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:15.002214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:15.084343 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:15.084314    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:15.786999 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:15.786946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:16.748787 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:16.748757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:17.791334 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:17.791302    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:18.831780 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:18.831750    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:18.931917 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:18.931887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:19.032732 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:19.032699    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:25.491819 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:25.491786    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:26.344443 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:26.344400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:29.401110 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:29.401080    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:29.498952 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:29.498922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:32.892873 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:32.892839    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:32.994317 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:32.994285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:33.094910 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:33.094880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:33.193629 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:33.193553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:33.810910 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:33.810880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:33.909778 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:33.909746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:34.018907 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:34.018879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:35.611199 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:35.611161    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:42.362485 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:42.362451    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:42.462653 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:42.462620    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:43.053104 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:43.053064    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:50.111783 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:50.111745    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:50.188258 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:50.188228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:50.406048 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:50.405948    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:50.833314 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:50.833282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:50.959757 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:50.959727    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:57.208586 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:57.208555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:57.874619 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:57.874581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:57.979247 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:57.979214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:58.075345 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:58.075312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:58.214089 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:58.214041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:59.120583 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:02:59.120545    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:00.019419 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:00.019388    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:00.937751 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:00.937723    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:01.803271 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:01.803231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:02.743857 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:02.743823    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:02.836364 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:02.836319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:02.947015 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:02.946958    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:03.479899 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.479869    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:03.586743 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.586714    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:03.782743 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.782661    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"]
Sep 16 19:03:03.785821 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.785805    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:03.789746 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.789728    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-050f3bde-dockercfg-fp4nx\""
Sep 16 19:03:03.790325 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.790308    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-050f3bde\""
Sep 16 19:03:03.797767 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.797744    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"]
Sep 16 19:03:03.833300 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.833269    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d667a7-e526-4abf-a7c4-319da00930d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs\" (UID: \"16d667a7-e526-4abf-a7c4-319da00930d9\") " pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:03.833482 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.833317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k97h7\" (UniqueName: \"kubernetes.io/projected/16d667a7-e526-4abf-a7c4-319da00930d9-kube-api-access-k97h7\") pod \"payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs\" (UID: \"16d667a7-e526-4abf-a7c4-319da00930d9\") " pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:03.934271 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.934233    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d667a7-e526-4abf-a7c4-319da00930d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs\" (UID: \"16d667a7-e526-4abf-a7c4-319da00930d9\") " pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:03.934675 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.934276    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k97h7\" (UniqueName: \"kubernetes.io/projected/16d667a7-e526-4abf-a7c4-319da00930d9-kube-api-access-k97h7\") pod \"payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs\" (UID: \"16d667a7-e526-4abf-a7c4-319da00930d9\") " pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:03.934950 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.934928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d667a7-e526-4abf-a7c4-319da00930d9-plugins-config-volume\") pod \"payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs\" (UID: \"16d667a7-e526-4abf-a7c4-319da00930d9\") " pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:03.944847 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:03.944826    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k97h7\" (UniqueName: \"kubernetes.io/projected/16d667a7-e526-4abf-a7c4-319da00930d9-kube-api-access-k97h7\") pod \"payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs\" (UID: \"16d667a7-e526-4abf-a7c4-319da00930d9\") " pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:04.096232 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:04.096140    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:04.375335 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:04.375314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"]
Sep 16 19:03:04.376306 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:03:04.376277    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d667a7_e526_4abf_a7c4_319da00930d9.slice/crio-9ba57e88149e50bcaa585a150c1ea256cc6cfed9a4f843d0da1f7c1cb1cccea7 WatchSource:0}: Error finding container 9ba57e88149e50bcaa585a150c1ea256cc6cfed9a4f843d0da1f7c1cb1cccea7: Status 404 returned error can't find the container with id 9ba57e88149e50bcaa585a150c1ea256cc6cfed9a4f843d0da1f7c1cb1cccea7
Sep 16 19:03:04.677820 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:04.677725    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs" event={"ID":"16d667a7-e526-4abf-a7c4-319da00930d9","Type":"ContainerStarted","Data":"c854d2e5dd40e6bc126fa1e4acd71482234137564735c504a36b1a395b2c10ce"}
Sep 16 19:03:04.677820 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:04.677806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs" event={"ID":"16d667a7-e526-4abf-a7c4-319da00930d9","Type":"ContainerStarted","Data":"9ba57e88149e50bcaa585a150c1ea256cc6cfed9a4f843d0da1f7c1cb1cccea7"}
Sep 16 19:03:04.678043 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:04.677932    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:04.705522 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:04.705470    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs" podStartSLOduration=1.705451659 podStartE2EDuration="1.705451659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:04.703739382 +0000 UTC m=+1417.071795297" watchObservedRunningTime="2026-09-16 19:03:04.705451659 +0000 UTC m=+1417.073507559"
Sep 16 19:03:15.686044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:15.686013    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:25.300214 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:25.300171    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:26.856858 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:26.856824    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:26.974698 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:26.974650    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:27.757602 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:27.757570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:27.968001 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:27.967956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:28.721605 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:28.721574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:28.808916 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:28.808883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:29.058087 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:29.057968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:29.204493 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:29.204462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:30.040170 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:30.040137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:30.889361 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:30.889329    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:36.245431 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:36.245400    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:36.317326 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:36.317295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:37.586853 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:37.586822    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:38.369488 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:38.369446    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:39.306134 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:39.306104    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:40.155126 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:40.155089    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:41.008626 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.008587    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:41.765897 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.765855    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"]
Sep 16 19:03:41.769327 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.769304    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.772671 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.772643    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9d2450\"/\"openshift-service-ca.crt\""
Sep 16 19:03:41.774198 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.774176    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9d2450\"/\"kube-root-ca.crt\""
Sep 16 19:03:41.774328 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.774207    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9d2450\"/\"explicit-model-9d2450-kserve-self-signed-certs\""
Sep 16 19:03:41.774328 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.774252    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9d2450\"/\"default-dockercfg-j7b8c\""
Sep 16 19:03:41.781358 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.781332    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"]
Sep 16 19:03:41.815044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.815011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:41.869187 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.869151    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-home\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.869187 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.869185    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-model-cache\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.869422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.869209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tmp-dir\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.869422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.869263    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tls-certs\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.869422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.869318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zvz\" (UniqueName: \"kubernetes.io/projected/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-kube-api-access-r5zvz\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.869422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.869377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-dshm\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.916757 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.916725    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:41.969926 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.969893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-dshm\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.970059 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.969955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-home\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.970059 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.969989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-model-cache\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.970059 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.970010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tmp-dir\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.970235 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.970136    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tls-certs\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.970235 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.970178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zvz\" (UniqueName: \"kubernetes.io/projected/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-kube-api-access-r5zvz\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.970456 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.970428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-home\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.970535 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.970452    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-model-cache\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.970535 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.970488    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tmp-dir\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.972303 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.972281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-dshm\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.972477 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.972460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tls-certs\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:41.980585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:41.980565    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zvz\" (UniqueName: \"kubernetes.io/projected/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-kube-api-access-r5zvz\") pod \"explicit-model-9d2450-kserve-668c9b48d5-mfcjq\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") " pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:42.080384 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:42.080296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:42.352160 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:42.352134    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"]
Sep 16 19:03:42.353935 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:03:42.353906    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5ea97d7_3593_4c9a_a1eb_0a6dc8221a9f.slice/crio-f2f1da09859bdbddf24e3e1dd83195e4819955f7214a1eca8436656b02c5f4d7 WatchSource:0}: Error finding container f2f1da09859bdbddf24e3e1dd83195e4819955f7214a1eca8436656b02c5f4d7: Status 404 returned error can't find the container with id f2f1da09859bdbddf24e3e1dd83195e4819955f7214a1eca8436656b02c5f4d7
Sep 16 19:03:42.652818 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:42.652695    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:42.860710 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:42.860676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq" event={"ID":"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f","Type":"ContainerStarted","Data":"1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b"}
Sep 16 19:03:42.860710 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:42.860716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq" event={"ID":"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f","Type":"ContainerStarted","Data":"f2f1da09859bdbddf24e3e1dd83195e4819955f7214a1eca8436656b02c5f4d7"}
Sep 16 19:03:42.884031 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:42.883955    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq" podStartSLOduration=1.8839366 podStartE2EDuration="1.8839366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:42.881700312 +0000 UTC m=+1455.249756231" watchObservedRunningTime="2026-09-16 19:03:42.8839366 +0000 UTC m=+1455.251992500"
Sep 16 19:03:43.430760 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:43.430718    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:44.920521 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:44.920491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:46.777227 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:46.777183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:47.646717 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.646688    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:47.694941 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.694902    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"]
Sep 16 19:03:47.695175 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.695143    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs" podUID="16d667a7-e526-4abf-a7c4-319da00930d9" containerName="payload-processing" containerID="cri-o://c854d2e5dd40e6bc126fa1e4acd71482234137564735c504a36b1a395b2c10ce" gracePeriod=30
Sep 16 19:03:47.726649 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.726608    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:47.784818 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.784792    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:47.871000 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.870962    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:47.886711 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.886650    2559 generic.go:356] "Generic (PLEG): container finished" podID="16d667a7-e526-4abf-a7c4-319da00930d9" containerID="c854d2e5dd40e6bc126fa1e4acd71482234137564735c504a36b1a395b2c10ce" exitCode=0
Sep 16 19:03:47.886853 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.886796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs" event={"ID":"16d667a7-e526-4abf-a7c4-319da00930d9","Type":"ContainerDied","Data":"c854d2e5dd40e6bc126fa1e4acd71482234137564735c504a36b1a395b2c10ce"}
Sep 16 19:03:47.928879 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.928801    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:47.950834 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:47.950809    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:48.011891 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.011858    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d667a7-e526-4abf-a7c4-319da00930d9-plugins-config-volume\") pod \"16d667a7-e526-4abf-a7c4-319da00930d9\" (UID: \"16d667a7-e526-4abf-a7c4-319da00930d9\") "
Sep 16 19:03:48.011891 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.011893    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k97h7\" (UniqueName: \"kubernetes.io/projected/16d667a7-e526-4abf-a7c4-319da00930d9-kube-api-access-k97h7\") pod \"16d667a7-e526-4abf-a7c4-319da00930d9\" (UID: \"16d667a7-e526-4abf-a7c4-319da00930d9\") "
Sep 16 19:03:48.012299 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.012275    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d667a7-e526-4abf-a7c4-319da00930d9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16d667a7-e526-4abf-a7c4-319da00930d9" (UID: "16d667a7-e526-4abf-a7c4-319da00930d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:48.013926 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.013903    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d667a7-e526-4abf-a7c4-319da00930d9-kube-api-access-k97h7" (OuterVolumeSpecName: "kube-api-access-k97h7") pod "16d667a7-e526-4abf-a7c4-319da00930d9" (UID: "16d667a7-e526-4abf-a7c4-319da00930d9"). InnerVolumeSpecName "kube-api-access-k97h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:48.112546 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.112508    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d667a7-e526-4abf-a7c4-319da00930d9-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:03:48.112546 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.112539    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k97h7\" (UniqueName: \"kubernetes.io/projected/16d667a7-e526-4abf-a7c4-319da00930d9-kube-api-access-k97h7\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:03:48.238041 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.238016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:48.339093 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.339061    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:48.893313 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.893283    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"
Sep 16 19:03:48.893313 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.893296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs" event={"ID":"16d667a7-e526-4abf-a7c4-319da00930d9","Type":"ContainerDied","Data":"9ba57e88149e50bcaa585a150c1ea256cc6cfed9a4f843d0da1f7c1cb1cccea7"}
Sep 16 19:03:48.893852 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.893365    2559 scope.go:117] "RemoveContainer" containerID="c854d2e5dd40e6bc126fa1e4acd71482234137564735c504a36b1a395b2c10ce"
Sep 16 19:03:48.914885 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.914849    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"]
Sep 16 19:03:48.921025 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:48.920999    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-050f3bde-648cbb48d5-b6vhs"]
Sep 16 19:03:49.130199 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:49.130166    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:50.223019 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:50.222954    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16d667a7-e526-4abf-a7c4-319da00930d9" path="/var/lib/kubelet/pods/16d667a7-e526-4abf-a7c4-319da00930d9/volumes"
Sep 16 19:03:50.336234 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:50.336204    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:51.086494 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:51.086462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:51.834325 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:51.834283    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:52.081179 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:52.081138    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:52.081179 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:52.081189    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:52.093642 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:52.093573    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:52.713621 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:52.713575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:52.923943 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:52.923914    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:03:55.176700 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.176667    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"]
Sep 16 19:03:55.177200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.177181    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d667a7-e526-4abf-a7c4-319da00930d9" containerName="payload-processing"
Sep 16 19:03:55.177276 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.177203    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d667a7-e526-4abf-a7c4-319da00930d9" containerName="payload-processing"
Sep 16 19:03:55.177335 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.177304    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d667a7-e526-4abf-a7c4-319da00930d9" containerName="payload-processing"
Sep 16 19:03:55.181898 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.181870    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.185016 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.184989    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1857c\"/\"distinct-w0-b1857c-kserve-self-signed-certs\""
Sep 16 19:03:55.194672 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.194643    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"]
Sep 16 19:03:55.223195 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.223163    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:55.261434 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.261401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-dshm\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.261434 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.261436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbnvk\" (UniqueName: \"kubernetes.io/projected/1d4437b7-681e-42f3-9839-f59e17b2b3ad-kube-api-access-mbnvk\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.261682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.261462    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-home\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.261682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.261552    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-model-cache\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.261682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.261608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tmp-dir\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.261682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.261650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tls-certs\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.362856 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.362819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-dshm\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.362856 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.362855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbnvk\" (UniqueName: \"kubernetes.io/projected/1d4437b7-681e-42f3-9839-f59e17b2b3ad-kube-api-access-mbnvk\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.363147 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.362881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-home\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.363147 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.362914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-model-cache\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.363147 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.362967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tmp-dir\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.363147 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.363026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tls-certs\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.363397 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.363367    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-model-cache\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.363466 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.363378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-home\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.363505 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.363464    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tmp-dir\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.365470 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.365448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tls-certs\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.365636 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.365621    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-dshm\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.376625 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.376593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbnvk\" (UniqueName: \"kubernetes.io/projected/1d4437b7-681e-42f3-9839-f59e17b2b3ad-kube-api-access-mbnvk\") pod \"distinct-w0-b1857c-kserve-5f69dc965f-djpzc\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.493172 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.493137    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:03:55.581926 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.581858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:55.694643 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.694612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:55.771461 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.771434    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"]
Sep 16 19:03:55.773815 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:03:55.773792    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d4437b7_681e_42f3_9839_f59e17b2b3ad.slice/crio-b724354564d88a4b21c5a201360014aebc8798574fa5bc4997723daab90ce572 WatchSource:0}: Error finding container b724354564d88a4b21c5a201360014aebc8798574fa5bc4997723daab90ce572: Status 404 returned error can't find the container with id b724354564d88a4b21c5a201360014aebc8798574fa5bc4997723daab90ce572
Sep 16 19:03:55.897808 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.897781    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:55.929506 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.929474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc" event={"ID":"1d4437b7-681e-42f3-9839-f59e17b2b3ad","Type":"ContainerStarted","Data":"3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6"}
Sep 16 19:03:55.929506 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.929512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc" event={"ID":"1d4437b7-681e-42f3-9839-f59e17b2b3ad","Type":"ContainerStarted","Data":"b724354564d88a4b21c5a201360014aebc8798574fa5bc4997723daab90ce572"}
Sep 16 19:03:55.948134 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:55.948086    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc" podStartSLOduration=0.948069806 podStartE2EDuration="948.069806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:55.946961411 +0000 UTC m=+1468.315017322" watchObservedRunningTime="2026-09-16 19:03:55.948069806 +0000 UTC m=+1468.316125758"
Sep 16 19:03:56.250141 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:56.250108    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:56.386892 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:56.386861    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:56.537943 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:03:56.537850    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:02.861068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:02.861033    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:02.935399 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:02.935370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:03.023228 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:03.023193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:03.227391 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:03.227361    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:05.493680 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:05.493644    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:04:05.494182 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:05.493927    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:04:05.506607 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:05.506583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:04:05.996057 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:05.996032    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:04:08.407108 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:08.407068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:10.162434 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:10.162401    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:10.264815 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:10.264742    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:11.091327 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:11.091298    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:11.388205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:11.388082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:11.887073 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:11.887031    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:13.837737 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:13.837691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:14.713315 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:14.713278    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:15.019433 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:15.019336    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:15.113598 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:15.113566    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:15.566685 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:15.566654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:15.661695 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:15.661665    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:16.537348 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:16.537313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:17.364335 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:17.364306    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:18.146473 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:18.146443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:19.116513 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.116480    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:19.179641 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.179608    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:19.289307 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.289275    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:19.629264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.629233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:19.695234 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.695185    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:19.759240 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.759204    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"]
Sep 16 19:04:19.762698 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.762676    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:19.765379 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.765357    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c20219-dockercfg-5qb6z\""
Sep 16 19:04:19.766608 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.766591    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c20219\""
Sep 16 19:04:19.772594 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.772572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"]
Sep 16 19:04:19.884828 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.884744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973095d-db08-41a4-af4d-74b4410606b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n\" (UID: \"d973095d-db08-41a4-af4d-74b4410606b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:19.884828 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.884787    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h42t\" (UniqueName: \"kubernetes.io/projected/d973095d-db08-41a4-af4d-74b4410606b9-kube-api-access-9h42t\") pod \"payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n\" (UID: \"d973095d-db08-41a4-af4d-74b4410606b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:19.985524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.985480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973095d-db08-41a4-af4d-74b4410606b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n\" (UID: \"d973095d-db08-41a4-af4d-74b4410606b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:19.985524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.985530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h42t\" (UniqueName: \"kubernetes.io/projected/d973095d-db08-41a4-af4d-74b4410606b9-kube-api-access-9h42t\") pod \"payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n\" (UID: \"d973095d-db08-41a4-af4d-74b4410606b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:19.986386 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.986359    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973095d-db08-41a4-af4d-74b4410606b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n\" (UID: \"d973095d-db08-41a4-af4d-74b4410606b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:19.994243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:19.994219    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h42t\" (UniqueName: \"kubernetes.io/projected/d973095d-db08-41a4-af4d-74b4410606b9-kube-api-access-9h42t\") pod \"payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n\" (UID: \"d973095d-db08-41a4-af4d-74b4410606b9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:20.037729 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:20.037674    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:20.074398 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:20.074365    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:20.348030 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:20.348002    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"]
Sep 16 19:04:20.350050 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:04:20.350018    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd973095d_db08_41a4_af4d_74b4410606b9.slice/crio-c54ceb4f40cd65857e901c11eeb347f0d075fef39edd4718786cb96e1295d66a WatchSource:0}: Error finding container c54ceb4f40cd65857e901c11eeb347f0d075fef39edd4718786cb96e1295d66a: Status 404 returned error can't find the container with id c54ceb4f40cd65857e901c11eeb347f0d075fef39edd4718786cb96e1295d66a
Sep 16 19:04:20.771269 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:20.771240    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"]
Sep 16 19:04:20.788003 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:20.787958    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:20.914932 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:20.914897    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:21.066582 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:21.066489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n" event={"ID":"d973095d-db08-41a4-af4d-74b4410606b9","Type":"ContainerStarted","Data":"8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc"}
Sep 16 19:04:21.066582 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:21.066552    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n" event={"ID":"d973095d-db08-41a4-af4d-74b4410606b9","Type":"ContainerStarted","Data":"c54ceb4f40cd65857e901c11eeb347f0d075fef39edd4718786cb96e1295d66a"}
Sep 16 19:04:21.066773 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:21.066626    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:21.087959 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:21.087905    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n" podStartSLOduration=2.087889808 podStartE2EDuration="2.087889808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:21.085615661 +0000 UTC m=+1493.453671581" watchObservedRunningTime="2026-09-16 19:04:21.087889808 +0000 UTC m=+1493.455945746"
Sep 16 19:04:23.934680 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:23.934645    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:24.594504 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:24.594465    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:24.632479 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:24.632445    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"]
Sep 16 19:04:24.632723 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:24.632695    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4" podUID="d7ed512e-9a9a-497b-a02c-54e59e47c95f" containerName="payload-processing" containerID="cri-o://e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1" gracePeriod=30
Sep 16 19:04:24.677713 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:24.677673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:24.749625 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:24.749590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:24.841686 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:24.841650    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:24.900294 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:24.900262    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:04:24.906471 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:24.906433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:25.029195 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.029159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ed512e-9a9a-497b-a02c-54e59e47c95f-plugins-config-volume\") pod \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\" (UID: \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\") "
Sep 16 19:04:25.029195 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.029199    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkpvk\" (UniqueName: \"kubernetes.io/projected/d7ed512e-9a9a-497b-a02c-54e59e47c95f-kube-api-access-xkpvk\") pod \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\" (UID: \"d7ed512e-9a9a-497b-a02c-54e59e47c95f\") "
Sep 16 19:04:25.029598 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.029507    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ed512e-9a9a-497b-a02c-54e59e47c95f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d7ed512e-9a9a-497b-a02c-54e59e47c95f" (UID: "d7ed512e-9a9a-497b-a02c-54e59e47c95f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:25.031360 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.031331    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ed512e-9a9a-497b-a02c-54e59e47c95f-kube-api-access-xkpvk" (OuterVolumeSpecName: "kube-api-access-xkpvk") pod "d7ed512e-9a9a-497b-a02c-54e59e47c95f" (UID: "d7ed512e-9a9a-497b-a02c-54e59e47c95f"). InnerVolumeSpecName "kube-api-access-xkpvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:25.087183 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.087149    2559 generic.go:356] "Generic (PLEG): container finished" podID="d7ed512e-9a9a-497b-a02c-54e59e47c95f" containerID="e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1" exitCode=0
Sep 16 19:04:25.087375 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.087195    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"
Sep 16 19:04:25.087375 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.087235    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4" event={"ID":"d7ed512e-9a9a-497b-a02c-54e59e47c95f","Type":"ContainerDied","Data":"e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1"}
Sep 16 19:04:25.087375 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.087330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4" event={"ID":"d7ed512e-9a9a-497b-a02c-54e59e47c95f","Type":"ContainerDied","Data":"6f456647b5042548e9711c9613d639d683d3cd10e8ecf46c898820b014e9e287"}
Sep 16 19:04:25.087375 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.087361    2559 scope.go:117] "RemoveContainer" containerID="e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1"
Sep 16 19:04:25.097436 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.097416    2559 scope.go:117] "RemoveContainer" containerID="e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1"
Sep 16 19:04:25.097709 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:04:25.097691    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1\": container with ID starting with e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1 not found: ID does not exist" containerID="e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1"
Sep 16 19:04:25.097848 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.097717    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1"} err="failed to get container status \"e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1\": rpc error: code = NotFound desc = could not find container \"e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1\": container with ID starting with e5de4d38f4429ab95004105be6e4a9e8c9c592d127c7994f4922136a70a2a9a1 not found: ID does not exist"
Sep 16 19:04:25.111823 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.111792    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"]
Sep 16 19:04:25.117458 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.117431    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-9d2450-7cf96876b9-tngv4"]
Sep 16 19:04:25.130563 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.130495    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7ed512e-9a9a-497b-a02c-54e59e47c95f-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:04:25.130563 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.130527    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpvk\" (UniqueName: \"kubernetes.io/projected/d7ed512e-9a9a-497b-a02c-54e59e47c95f-kube-api-access-xkpvk\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:04:25.198559 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.198517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:25.300816 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:25.300778    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:26.016697 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:26.016664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:26.222506 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:26.222471    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ed512e-9a9a-497b-a02c-54e59e47c95f" path="/var/lib/kubelet/pods/d7ed512e-9a9a-497b-a02c-54e59e47c95f/volumes"
Sep 16 19:04:26.914272 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:26.914236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:27.826570 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:27.826537    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:28.071270 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:28.071239    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:04:28.071941 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:28.071920    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:04:28.808625 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:28.808592    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:29.712212 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:29.712178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:30.284023 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:30.283980    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:31.213414 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:31.213381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:32.073173 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:32.073133    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:04:32.132924 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:32.132895    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:33.020715 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:33.020687    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:33.804364 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:33.804335    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:34.588444 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:34.588413    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:34.793609 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:34.793574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:35.143632 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:35.143602    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:35.399184 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:35.399109    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:35.518225 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:35.518194    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:35.790984 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:35.790930    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq" podUID="a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" containerName="main" containerID="cri-o://1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b" gracePeriod=45
Sep 16 19:04:36.031349 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.031320    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:36.043268 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.043220    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:04:36.130505 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130474    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tmp-dir\") pod \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") "
Sep 16 19:04:36.130666 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130533    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-home\") pod \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") "
Sep 16 19:04:36.130666 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130607    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5zvz\" (UniqueName: \"kubernetes.io/projected/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-kube-api-access-r5zvz\") pod \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") "
Sep 16 19:04:36.130666 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130636    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-model-cache\") pod \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") "
Sep 16 19:04:36.130666 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130663    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tls-certs\") pod \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") "
Sep 16 19:04:36.130886 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130769    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" (UID: "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:36.130886 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130790    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-dshm\") pod \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\" (UID: \"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f\") "
Sep 16 19:04:36.130886 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130830    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-home" (OuterVolumeSpecName: "home") pod "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" (UID: "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:36.131077 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.130894    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" (UID: "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:36.131160 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.131140    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:04:36.131223 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.131165    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:04:36.131223 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.131174    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:04:36.132933 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.132904    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" (UID: "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:36.133049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.132988    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-dshm" (OuterVolumeSpecName: "dshm") pod "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" (UID: "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:36.133049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.133028    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-kube-api-access-r5zvz" (OuterVolumeSpecName: "kube-api-access-r5zvz") pod "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" (UID: "a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f"). InnerVolumeSpecName "kube-api-access-r5zvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:36.145119 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.145093    2559 generic.go:356] "Generic (PLEG): container finished" podID="a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" containerID="1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b" exitCode=0
Sep 16 19:04:36.145248 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.145134    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"
Sep 16 19:04:36.145248 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.145178    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq" event={"ID":"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f","Type":"ContainerDied","Data":"1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b"}
Sep 16 19:04:36.145248 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.145219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq" event={"ID":"a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f","Type":"ContainerDied","Data":"f2f1da09859bdbddf24e3e1dd83195e4819955f7214a1eca8436656b02c5f4d7"}
Sep 16 19:04:36.145248 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.145238    2559 scope.go:117] "RemoveContainer" containerID="1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b"
Sep 16 19:04:36.154262 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.154244    2559 scope.go:117] "RemoveContainer" containerID="1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b"
Sep 16 19:04:36.154553 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:04:36.154533    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b\": container with ID starting with 1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b not found: ID does not exist" containerID="1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b"
Sep 16 19:04:36.154605 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.154560    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b"} err="failed to get container status \"1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b\": rpc error: code = NotFound desc = could not find container \"1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b\": container with ID starting with 1f70b31b5cb2eb97728544e8987a8079b5cac7ceb9ba00ce13d88776bf694b4b not found: ID does not exist"
Sep 16 19:04:36.171332 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.171311    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"]
Sep 16 19:04:36.172929 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.172908    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-9d2450/explicit-model-9d2450-kserve-668c9b48d5-mfcjq"]
Sep 16 19:04:36.223368 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.223333    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" path="/var/lib/kubelet/pods/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f/volumes"
Sep 16 19:04:36.231727 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.231692    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zvz\" (UniqueName: \"kubernetes.io/projected/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-kube-api-access-r5zvz\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:04:36.231727 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.231720    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:04:36.231727 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:36.231734    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:04:38.472394 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:38.472360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:38.684291 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:38.684253    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:39.214267 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:39.214241    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:39.322338 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:39.322309    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:39.483831 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:39.483755    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:40.614273 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:40.614236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:41.653003 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:41.652957    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:42.762309 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:42.762260    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:43.833212 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:43.833179    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:44.826554 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:44.826521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:44.924103 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:44.924073    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:45.022601 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.022573    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:45.400769 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.400741    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:45.507337 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.507305    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:45.533435 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.533396    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"]
Sep 16 19:04:45.533813 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.533795    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" containerName="main"
Sep 16 19:04:45.533892 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.533816    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" containerName="main"
Sep 16 19:04:45.533892 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.533844    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ed512e-9a9a-497b-a02c-54e59e47c95f" containerName="payload-processing"
Sep 16 19:04:45.533892 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.533853    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ed512e-9a9a-497b-a02c-54e59e47c95f" containerName="payload-processing"
Sep 16 19:04:45.534084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.533931    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ed512e-9a9a-497b-a02c-54e59e47c95f" containerName="payload-processing"
Sep 16 19:04:45.534084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.533947    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ea97d7-3593-4c9a-a1eb-0a6dc8221a9f" containerName="main"
Sep 16 19:04:45.538224 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.538202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:45.544742 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.544722    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-58bf51\""
Sep 16 19:04:45.558276 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.558247    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"]
Sep 16 19:04:45.607964 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.607919    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtxlc\" (UniqueName: \"kubernetes.io/projected/1a30f659-1519-496e-b079-fd318ab77dad-kube-api-access-rtxlc\") pod \"payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6\" (UID: \"1a30f659-1519-496e-b079-fd318ab77dad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:45.608182 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.608059    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a30f659-1519-496e-b079-fd318ab77dad-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6\" (UID: \"1a30f659-1519-496e-b079-fd318ab77dad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:45.708788 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.708754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a30f659-1519-496e-b079-fd318ab77dad-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6\" (UID: \"1a30f659-1519-496e-b079-fd318ab77dad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:45.708949 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.708819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtxlc\" (UniqueName: \"kubernetes.io/projected/1a30f659-1519-496e-b079-fd318ab77dad-kube-api-access-rtxlc\") pod \"payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6\" (UID: \"1a30f659-1519-496e-b079-fd318ab77dad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:45.709537 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.709518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a30f659-1519-496e-b079-fd318ab77dad-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6\" (UID: \"1a30f659-1519-496e-b079-fd318ab77dad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:45.717868 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.717833    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtxlc\" (UniqueName: \"kubernetes.io/projected/1a30f659-1519-496e-b079-fd318ab77dad-kube-api-access-rtxlc\") pod \"payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6\" (UID: \"1a30f659-1519-496e-b079-fd318ab77dad\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:45.848897 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:45.848856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:46.125401 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:46.125376    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"]
Sep 16 19:04:46.128073 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:04:46.128043    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a30f659_1519_496e_b079_fd318ab77dad.slice/crio-204891da5c4a13ee4d5d816299389ca420c911b4ccebd2f426322ddae49e7e8f WatchSource:0}: Error finding container 204891da5c4a13ee4d5d816299389ca420c911b4ccebd2f426322ddae49e7e8f: Status 404 returned error can't find the container with id 204891da5c4a13ee4d5d816299389ca420c911b4ccebd2f426322ddae49e7e8f
Sep 16 19:04:46.195654 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:46.195628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6" event={"ID":"1a30f659-1519-496e-b079-fd318ab77dad","Type":"ContainerStarted","Data":"204891da5c4a13ee4d5d816299389ca420c911b4ccebd2f426322ddae49e7e8f"}
Sep 16 19:04:47.202367 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:47.202332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6" event={"ID":"1a30f659-1519-496e-b079-fd318ab77dad","Type":"ContainerStarted","Data":"b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714"}
Sep 16 19:04:47.202761 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:47.202439    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:04:47.222486 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:47.222443    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6" podStartSLOduration=2.222430006 podStartE2EDuration="2.222430006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:47.221137525 +0000 UTC m=+1519.589193478" watchObservedRunningTime="2026-09-16 19:04:47.222430006 +0000 UTC m=+1519.590485906"
Sep 16 19:04:58.208921 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:04:58.208888    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:05:00.288741 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:00.288707    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:00.389041 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:00.388955    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:08.338776 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:08.338742    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:08.625858 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:08.625766    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:08.687665 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:08.687619    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:09.081415 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:09.081380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:09.264634 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:09.264604    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:09.390384 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:09.390295    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:09.610624 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:09.610595    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:09.803828 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:09.803799    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:09.989281 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:09.989247    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7223-162178"]
Sep 16 19:05:09.992402 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:09.992387    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7223-162178"
Sep 16 19:05:09.995225 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:09.995203    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q2jrz\""
Sep 16 19:05:10.000186 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.000170    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7223-162178"]
Sep 16 19:05:10.134126 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.134028    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdfz\" (UniqueName: \"kubernetes.io/projected/a8df5455-7303-48fd-806c-d9c62528b172-kube-api-access-hmdfz\") pod \"test-curl-7223-162178\" (UID: \"a8df5455-7303-48fd-806c-d9c62528b172\") " pod="opendatahub/test-curl-7223-162178"
Sep 16 19:05:10.234648 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.234616    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdfz\" (UniqueName: \"kubernetes.io/projected/a8df5455-7303-48fd-806c-d9c62528b172-kube-api-access-hmdfz\") pod \"test-curl-7223-162178\" (UID: \"a8df5455-7303-48fd-806c-d9c62528b172\") " pod="opendatahub/test-curl-7223-162178"
Sep 16 19:05:10.249253 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.249221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdfz\" (UniqueName: \"kubernetes.io/projected/a8df5455-7303-48fd-806c-d9c62528b172-kube-api-access-hmdfz\") pod \"test-curl-7223-162178\" (UID: \"a8df5455-7303-48fd-806c-d9c62528b172\") " pod="opendatahub/test-curl-7223-162178"
Sep 16 19:05:10.302851 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.302804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7223-162178"
Sep 16 19:05:10.578806 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.578660    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7223-162178"]
Sep 16 19:05:10.581291 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:05:10.581264    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8df5455_7303_48fd_806c_d9c62528b172.slice/crio-9db063cd41e4b96f3db3705e458cc3e7b6fc1c9dac8fdad6fde050d900fed7c4 WatchSource:0}: Error finding container 9db063cd41e4b96f3db3705e458cc3e7b6fc1c9dac8fdad6fde050d900fed7c4: Status 404 returned error can't find the container with id 9db063cd41e4b96f3db3705e458cc3e7b6fc1c9dac8fdad6fde050d900fed7c4
Sep 16 19:05:10.583207 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.583186    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:05:10.702541 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.702504    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:10.787499 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.787458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:10.913524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:10.913433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:11.315629 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:11.315596    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7223-162178" event={"ID":"a8df5455-7303-48fd-806c-d9c62528b172","Type":"ContainerStarted","Data":"b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033"}
Sep 16 19:05:11.315629 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:11.315638    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7223-162178" event={"ID":"a8df5455-7303-48fd-806c-d9c62528b172","Type":"ContainerStarted","Data":"9db063cd41e4b96f3db3705e458cc3e7b6fc1c9dac8fdad6fde050d900fed7c4"}
Sep 16 19:05:11.336905 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:11.336856    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7223-162178" podStartSLOduration=1.6858275040000001 podStartE2EDuration="2.336841862s" totalImagesPullingTime="651.014358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:11.33604223 +0000 UTC m=+1543.704098146" watchObservedRunningTime="2026-09-16 19:05:11.336841862 +0000 UTC m=+1543.704897763"
Sep 16 19:05:11.618502 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:11.618469    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:11.992896 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:11.992862    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7223-162178"]
Sep 16 19:05:11.995289 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:11.995258    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7223-162178"]
Sep 16 19:05:12.110955 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.110874    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7223-1d73d6"]
Sep 16 19:05:12.111282 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.111268    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8df5455-7303-48fd-806c-d9c62528b172" containerName="test-curl-7223-162178"
Sep 16 19:05:12.111330 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.111284    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8df5455-7303-48fd-806c-d9c62528b172" containerName="test-curl-7223-162178"
Sep 16 19:05:12.111369 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.111344    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8df5455-7303-48fd-806c-d9c62528b172" containerName="test-curl-7223-162178"
Sep 16 19:05:12.114311 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.114289    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7223-1d73d6"
Sep 16 19:05:12.120512 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.120485    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7223-1d73d6"]
Sep 16 19:05:12.253671 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.253639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnlv8\" (UniqueName: \"kubernetes.io/projected/88899ba5-af60-4710-8275-3f6460d6af91-kube-api-access-mnlv8\") pod \"test-curl-7223-1d73d6\" (UID: \"88899ba5-af60-4710-8275-3f6460d6af91\") " pod="opendatahub/test-curl-7223-1d73d6"
Sep 16 19:05:12.320855 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:05:12.320826    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7223-162178" container="test-curl-7223-162178"
Sep 16 19:05:12.354598 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.354562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnlv8\" (UniqueName: \"kubernetes.io/projected/88899ba5-af60-4710-8275-3f6460d6af91-kube-api-access-mnlv8\") pod \"test-curl-7223-1d73d6\" (UID: \"88899ba5-af60-4710-8275-3f6460d6af91\") " pod="opendatahub/test-curl-7223-1d73d6"
Sep 16 19:05:12.364103 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.364035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnlv8\" (UniqueName: \"kubernetes.io/projected/88899ba5-af60-4710-8275-3f6460d6af91-kube-api-access-mnlv8\") pod \"test-curl-7223-1d73d6\" (UID: \"88899ba5-af60-4710-8275-3f6460d6af91\") " pod="opendatahub/test-curl-7223-1d73d6"
Sep 16 19:05:12.425176 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.425141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7223-1d73d6"
Sep 16 19:05:12.528785 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.528750    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:12.705023 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:12.704986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7223-1d73d6"]
Sep 16 19:05:12.707411 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:05:12.707376    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88899ba5_af60_4710_8275_3f6460d6af91.slice/crio-4f0789b85445ad5405c94564fa361c18ef80c5c867ee1c8516c9a7b4f6dc819c WatchSource:0}: Error finding container 4f0789b85445ad5405c94564fa361c18ef80c5c867ee1c8516c9a7b4f6dc819c: Status 404 returned error can't find the container with id 4f0789b85445ad5405c94564fa361c18ef80c5c867ee1c8516c9a7b4f6dc819c
Sep 16 19:05:13.327797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:13.327753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7223-1d73d6" event={"ID":"88899ba5-af60-4710-8275-3f6460d6af91","Type":"ContainerStarted","Data":"c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d"}
Sep 16 19:05:13.328012 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:13.327805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7223-1d73d6" event={"ID":"88899ba5-af60-4710-8275-3f6460d6af91","Type":"ContainerStarted","Data":"4f0789b85445ad5405c94564fa361c18ef80c5c867ee1c8516c9a7b4f6dc819c"}
Sep 16 19:05:13.328012 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:13.327847    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7223-162178" podUID="a8df5455-7303-48fd-806c-d9c62528b172" containerName="test-curl-7223-162178" containerID="cri-o://b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033" gracePeriod=2
Sep 16 19:05:13.346223 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:13.346177    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7223-1d73d6" podStartSLOduration=1.346161837 podStartE2EDuration="1.346161837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:13.345262763 +0000 UTC m=+1545.713318663" watchObservedRunningTime="2026-09-16 19:05:13.346161837 +0000 UTC m=+1545.714217737"
Sep 16 19:05:13.557671 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:13.557643    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:13.996454 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:13.996417    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7223-1d73d6"]
Sep 16 19:05:14.000634 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:14.000606    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7223-1d73d6"]
Sep 16 19:05:14.332359 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:05:14.332326    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7223-1d73d6" container="test-curl-7223-1d73d6"
Sep 16 19:05:14.420809 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:14.420778    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:15.337275 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:15.337238    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7223-1d73d6" podUID="88899ba5-af60-4710-8275-3f6460d6af91" containerName="test-curl-7223-1d73d6" containerID="cri-o://c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d" gracePeriod=2
Sep 16 19:05:15.466215 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:15.466191    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7223-162178"
Sep 16 19:05:15.528750 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:15.528715    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:15.584212 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:15.584129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmdfz\" (UniqueName: \"kubernetes.io/projected/a8df5455-7303-48fd-806c-d9c62528b172-kube-api-access-hmdfz\") pod \"a8df5455-7303-48fd-806c-d9c62528b172\" (UID: \"a8df5455-7303-48fd-806c-d9c62528b172\") "
Sep 16 19:05:15.586360 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:15.586339    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8df5455-7303-48fd-806c-d9c62528b172-kube-api-access-hmdfz" (OuterVolumeSpecName: "kube-api-access-hmdfz") pod "a8df5455-7303-48fd-806c-d9c62528b172" (UID: "a8df5455-7303-48fd-806c-d9c62528b172"). InnerVolumeSpecName "kube-api-access-hmdfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:15.685735 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:15.685696    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmdfz\" (UniqueName: \"kubernetes.io/projected/a8df5455-7303-48fd-806c-d9c62528b172-kube-api-access-hmdfz\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:05:16.222956 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:16.222916    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8df5455-7303-48fd-806c-d9c62528b172" path="/var/lib/kubelet/pods/a8df5455-7303-48fd-806c-d9c62528b172/volumes"
Sep 16 19:05:16.332097 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:16.332064    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:16.343474 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:16.343439    2559 generic.go:356] "Generic (PLEG): container finished" podID="a8df5455-7303-48fd-806c-d9c62528b172" containerID="b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033" exitCode=137
Sep 16 19:05:16.343853 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:16.343484    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7223-162178"
Sep 16 19:05:16.343853 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:16.343533    2559 scope.go:117] "RemoveContainer" containerID="b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033"
Sep 16 19:05:16.351881 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:16.351861    2559 scope.go:117] "RemoveContainer" containerID="b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033"
Sep 16 19:05:16.352163 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:05:16.352142    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033\": container with ID starting with b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033 not found: ID does not exist" containerID="b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033"
Sep 16 19:05:16.352212 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:16.352172    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033"} err="failed to get container status \"b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033\": rpc error: code = NotFound desc = could not find container \"b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033\": container with ID starting with b5feb3d9237b88e445fe1aa5004b7666664bf4e11f3fa6af977dd7383515a033 not found: ID does not exist"
Sep 16 19:05:17.276675 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.276639    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"]
Sep 16 19:05:17.277005 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.276992    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88899ba5-af60-4710-8275-3f6460d6af91" containerName="test-curl-7223-1d73d6"
Sep 16 19:05:17.277005 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.277006    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="88899ba5-af60-4710-8275-3f6460d6af91" containerName="test-curl-7223-1d73d6"
Sep 16 19:05:17.277098 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.277068    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="88899ba5-af60-4710-8275-3f6460d6af91" containerName="test-curl-7223-1d73d6"
Sep 16 19:05:17.281223 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.281207    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.284258 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.284239    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1857c\"/\"distinct-2-w0-b1857c-kserve-self-signed-certs\""
Sep 16 19:05:17.291632 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.291610    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"]
Sep 16 19:05:17.382233 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.382202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:17.402528 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.402496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-home\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.402668 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.402533    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-model-cache\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.402668 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.402601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-dshm\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.402752 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.402682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bc1e47-8865-4673-89b1-63751f3e2e98-tls-certs\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.402752 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.402740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-tmp-dir\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.402838 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.402789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92w62\" (UniqueName: \"kubernetes.io/projected/f4bc1e47-8865-4673-89b1-63751f3e2e98-kube-api-access-92w62\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.484741 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.484716    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7223-1d73d6"
Sep 16 19:05:17.503510 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.503486    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-home\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.503623 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.503515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-model-cache\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.503623 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.503543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-dshm\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.503623 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.503562    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bc1e47-8865-4673-89b1-63751f3e2e98-tls-certs\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.503623 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.503596    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-tmp-dir\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.503846 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.503633    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92w62\" (UniqueName: \"kubernetes.io/projected/f4bc1e47-8865-4673-89b1-63751f3e2e98-kube-api-access-92w62\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.503921 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.503898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-home\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.504012 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.503960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-model-cache\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.504061 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.504044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-tmp-dir\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.505822 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.505794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-dshm\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.506014 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.505999    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bc1e47-8865-4673-89b1-63751f3e2e98-tls-certs\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.514434 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.514409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92w62\" (UniqueName: \"kubernetes.io/projected/f4bc1e47-8865-4673-89b1-63751f3e2e98-kube-api-access-92w62\") pod \"distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") " pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.590652 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.590578    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:17.604607 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.604582    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnlv8\" (UniqueName: \"kubernetes.io/projected/88899ba5-af60-4710-8275-3f6460d6af91-kube-api-access-mnlv8\") pod \"88899ba5-af60-4710-8275-3f6460d6af91\" (UID: \"88899ba5-af60-4710-8275-3f6460d6af91\") "
Sep 16 19:05:17.606478 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.606453    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88899ba5-af60-4710-8275-3f6460d6af91-kube-api-access-mnlv8" (OuterVolumeSpecName: "kube-api-access-mnlv8") pod "88899ba5-af60-4710-8275-3f6460d6af91" (UID: "88899ba5-af60-4710-8275-3f6460d6af91"). InnerVolumeSpecName "kube-api-access-mnlv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:17.705603 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.705563    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnlv8\" (UniqueName: \"kubernetes.io/projected/88899ba5-af60-4710-8275-3f6460d6af91-kube-api-access-mnlv8\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:05:17.873607 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:17.873541    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"]
Sep 16 19:05:17.875744 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:05:17.875715    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4bc1e47_8865_4673_89b1_63751f3e2e98.slice/crio-a5572f888866219ac57acd40cbc8c9ee362dedacc560f5971656382129463576 WatchSource:0}: Error finding container a5572f888866219ac57acd40cbc8c9ee362dedacc560f5971656382129463576: Status 404 returned error can't find the container with id a5572f888866219ac57acd40cbc8c9ee362dedacc560f5971656382129463576
Sep 16 19:05:18.224248 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.224221    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88899ba5-af60-4710-8275-3f6460d6af91" path="/var/lib/kubelet/pods/88899ba5-af60-4710-8275-3f6460d6af91/volumes"
Sep 16 19:05:18.358268 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.358221    2559 generic.go:356] "Generic (PLEG): container finished" podID="88899ba5-af60-4710-8275-3f6460d6af91" containerID="c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d" exitCode=137
Sep 16 19:05:18.358427 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.358281    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7223-1d73d6"
Sep 16 19:05:18.358427 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.358325    2559 scope.go:117] "RemoveContainer" containerID="c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d"
Sep 16 19:05:18.360211 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.360185    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6" event={"ID":"f4bc1e47-8865-4673-89b1-63751f3e2e98","Type":"ContainerStarted","Data":"ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81"}
Sep 16 19:05:18.360332 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.360227    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6" event={"ID":"f4bc1e47-8865-4673-89b1-63751f3e2e98","Type":"ContainerStarted","Data":"a5572f888866219ac57acd40cbc8c9ee362dedacc560f5971656382129463576"}
Sep 16 19:05:18.367108 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.367094    2559 scope.go:117] "RemoveContainer" containerID="c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d"
Sep 16 19:05:18.367384 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:05:18.367363    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d\": container with ID starting with c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d not found: ID does not exist" containerID="c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d"
Sep 16 19:05:18.367456 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.367395    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d"} err="failed to get container status \"c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d\": rpc error: code = NotFound desc = could not find container \"c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d\": container with ID starting with c00bc89cb39555dcbd5f702ff8950a550bc32e1cbc3c3973eb81fc2350099d7d not found: ID does not exist"
Sep 16 19:05:18.380130 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:18.380091    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6" podStartSLOduration=1.380078569 podStartE2EDuration="1.380078569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:18.37974649 +0000 UTC m=+1550.747802390" watchObservedRunningTime="2026-09-16 19:05:18.380078569 +0000 UTC m=+1550.748134472"
Sep 16 19:05:19.813270 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:19.813240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:20.546361 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.546276    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:20.567746 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.567714    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"]
Sep 16 19:05:20.575517 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.575488    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec32e7-29826420-dxqfd"]
Sep 16 19:05:20.595452 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.595423    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"]
Sep 16 19:05:20.595671 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.595643    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml" podUID="d2ff2a40-eaef-43db-90dd-3cd36fb512bf" containerName="payload-processing" containerID="cri-o://7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179" gracePeriod=30
Sep 16 19:05:20.603327 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.603298    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"]
Sep 16 19:05:20.603735 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.603551    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn" podUID="4e388f47-3047-42a6-a93f-b84b3288a584" containerName="payload-pre-processing" containerID="cri-o://f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2" gracePeriod=30
Sep 16 19:05:20.651561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.651530    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:20.843324 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.843290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:20.882142 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.882114    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 19:05:20.884176 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.884158    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 19:05:20.930542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.930513    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:20.932657 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.932636    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk9qv\" (UniqueName: \"kubernetes.io/projected/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-kube-api-access-kk9qv\") pod \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\" (UID: \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\") "
Sep 16 19:05:20.932824 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.932687    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-plugins-config-volume\") pod \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\" (UID: \"d2ff2a40-eaef-43db-90dd-3cd36fb512bf\") "
Sep 16 19:05:20.932824 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.932773    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e388f47-3047-42a6-a93f-b84b3288a584-plugins-config-volume\") pod \"4e388f47-3047-42a6-a93f-b84b3288a584\" (UID: \"4e388f47-3047-42a6-a93f-b84b3288a584\") "
Sep 16 19:05:20.932824 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.932816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lbzz\" (UniqueName: \"kubernetes.io/projected/4e388f47-3047-42a6-a93f-b84b3288a584-kube-api-access-5lbzz\") pod \"4e388f47-3047-42a6-a93f-b84b3288a584\" (UID: \"4e388f47-3047-42a6-a93f-b84b3288a584\") "
Sep 16 19:05:20.933150 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.933123    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2ff2a40-eaef-43db-90dd-3cd36fb512bf" (UID: "d2ff2a40-eaef-43db-90dd-3cd36fb512bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:20.933232 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.933152    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e388f47-3047-42a6-a93f-b84b3288a584-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e388f47-3047-42a6-a93f-b84b3288a584" (UID: "4e388f47-3047-42a6-a93f-b84b3288a584"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:05:20.935198 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.935168    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e388f47-3047-42a6-a93f-b84b3288a584-kube-api-access-5lbzz" (OuterVolumeSpecName: "kube-api-access-5lbzz") pod "4e388f47-3047-42a6-a93f-b84b3288a584" (UID: "4e388f47-3047-42a6-a93f-b84b3288a584"). InnerVolumeSpecName "kube-api-access-5lbzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:20.935335 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:20.935276    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-kube-api-access-kk9qv" (OuterVolumeSpecName: "kube-api-access-kk9qv") pod "d2ff2a40-eaef-43db-90dd-3cd36fb512bf" (UID: "d2ff2a40-eaef-43db-90dd-3cd36fb512bf"). InnerVolumeSpecName "kube-api-access-kk9qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:21.033423 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.033383    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e388f47-3047-42a6-a93f-b84b3288a584-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:05:21.033423 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.033416    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lbzz\" (UniqueName: \"kubernetes.io/projected/4e388f47-3047-42a6-a93f-b84b3288a584-kube-api-access-5lbzz\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:05:21.033423 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.033425    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk9qv\" (UniqueName: \"kubernetes.io/projected/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-kube-api-access-kk9qv\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:05:21.033665 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.033434    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2ff2a40-eaef-43db-90dd-3cd36fb512bf-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:05:21.149670 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.149594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:21.237929 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.237889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:21.328184 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.328149    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:21.377513 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.377473    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2ff2a40-eaef-43db-90dd-3cd36fb512bf" containerID="7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179" exitCode=0
Sep 16 19:05:21.377513 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.377514    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"
Sep 16 19:05:21.377760 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.377560    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml" event={"ID":"d2ff2a40-eaef-43db-90dd-3cd36fb512bf","Type":"ContainerDied","Data":"7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179"}
Sep 16 19:05:21.377760 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.377617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml" event={"ID":"d2ff2a40-eaef-43db-90dd-3cd36fb512bf","Type":"ContainerDied","Data":"fc5328e98ef6eb3f175c222f091cbede733751f71db5b34dd2f7a668fbeed228"}
Sep 16 19:05:21.377760 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.377648    2559 scope.go:117] "RemoveContainer" containerID="7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179"
Sep 16 19:05:21.379040 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.379014    2559 generic.go:356] "Generic (PLEG): container finished" podID="4e388f47-3047-42a6-a93f-b84b3288a584" containerID="f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2" exitCode=0
Sep 16 19:05:21.379159 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.379069    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"
Sep 16 19:05:21.379159 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.379092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn" event={"ID":"4e388f47-3047-42a6-a93f-b84b3288a584","Type":"ContainerDied","Data":"f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2"}
Sep 16 19:05:21.379159 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.379143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn" event={"ID":"4e388f47-3047-42a6-a93f-b84b3288a584","Type":"ContainerDied","Data":"7716a2cabd5bb58c877f3a86c02529a8fdb9fae5eb616507732e1d446f1d3f86"}
Sep 16 19:05:21.387644 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.387625    2559 scope.go:117] "RemoveContainer" containerID="7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179"
Sep 16 19:05:21.387894 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:05:21.387875    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179\": container with ID starting with 7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179 not found: ID does not exist" containerID="7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179"
Sep 16 19:05:21.387988 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.387900    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179"} err="failed to get container status \"7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179\": rpc error: code = NotFound desc = could not find container \"7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179\": container with ID starting with 7551c29a0413c91a04fbaa66424f0e827f45b84b7403d290ad929ef6aa6cf179 not found: ID does not exist"
Sep 16 19:05:21.387988 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.387917    2559 scope.go:117] "RemoveContainer" containerID="f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2"
Sep 16 19:05:21.395617 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.395597    2559 scope.go:117] "RemoveContainer" containerID="f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2"
Sep 16 19:05:21.395873 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:05:21.395855    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2\": container with ID starting with f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2 not found: ID does not exist" containerID="f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2"
Sep 16 19:05:21.395922 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.395879    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2"} err="failed to get container status \"f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2\": rpc error: code = NotFound desc = could not find container \"f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2\": container with ID starting with f41bb0fc14e26ddd04d54637f49a72f8d69e13ab5d6f53bf3696b4933dfbd2e2 not found: ID does not exist"
Sep 16 19:05:21.405044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.404963    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"]
Sep 16 19:05:21.407602 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.407581    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec32e7-69979d9c79-mg8ml"]
Sep 16 19:05:21.419816 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.419789    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"]
Sep 16 19:05:21.421488 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.421469    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec32e7-689f665f58-zd8gn"]
Sep 16 19:05:21.426494 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:21.426477    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:22.223993 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:22.223936    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d" path="/var/lib/kubelet/pods/1d3da3c8-e3d5-4873-80f6-e2947b9c8a3d/volumes"
Sep 16 19:05:22.224481 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:22.224437    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e388f47-3047-42a6-a93f-b84b3288a584" path="/var/lib/kubelet/pods/4e388f47-3047-42a6-a93f-b84b3288a584/volumes"
Sep 16 19:05:22.224844 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:22.224824    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ff2a40-eaef-43db-90dd-3cd36fb512bf" path="/var/lib/kubelet/pods/d2ff2a40-eaef-43db-90dd-3cd36fb512bf/volumes"
Sep 16 19:05:22.415137 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:22.415105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:22.787082 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:22.787050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:23.454653 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:23.454621    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:24.568637 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:24.568599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:24.716670 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:24.716627    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:24.847084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:24.847013    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:25.852930 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:25.852893    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:26.761982 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:26.761953    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:27.591427 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:27.591388    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:27.591898 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:27.591449    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:27.604176 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:27.604143    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:27.752526 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:27.752487    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:27.835821 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:27.835794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:27.953219 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:27.953107    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:28.134809 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.134776    2559 scope.go:117] "RemoveContainer" containerID="f5db63d712b54d679dbc4b074dbd584554b2dd517ec2a45766734f31732244ca"
Sep 16 19:05:28.143687 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.143668    2559 scope.go:117] "RemoveContainer" containerID="ae2ab5c05456de5e2bc7a915d30278684aab49e79230d7e7dd8c77aff47c6fb3"
Sep 16 19:05:28.346609 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.346577    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:28.434963 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.434928    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:05:28.447308 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.447285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:28.474638 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.474613    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"]
Sep 16 19:05:28.475009 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.474992    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ff2a40-eaef-43db-90dd-3cd36fb512bf" containerName="payload-processing"
Sep 16 19:05:28.475059 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.475013    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ff2a40-eaef-43db-90dd-3cd36fb512bf" containerName="payload-processing"
Sep 16 19:05:28.475059 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.475026    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e388f47-3047-42a6-a93f-b84b3288a584" containerName="payload-pre-processing"
Sep 16 19:05:28.475059 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.475035    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e388f47-3047-42a6-a93f-b84b3288a584" containerName="payload-pre-processing"
Sep 16 19:05:28.475162 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.475131    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ff2a40-eaef-43db-90dd-3cd36fb512bf" containerName="payload-processing"
Sep 16 19:05:28.475162 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.475146    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e388f47-3047-42a6-a93f-b84b3288a584" containerName="payload-pre-processing"
Sep 16 19:05:28.479712 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.479690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:28.482749 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.482720    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d929c607\""
Sep 16 19:05:28.491888 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.491860    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"]
Sep 16 19:05:28.492734 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.492712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccdbed-282e-4205-a4f3-c535e41237ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5\" (UID: \"45ccdbed-282e-4205-a4f3-c535e41237ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:28.492822 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.492786    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5xn\" (UniqueName: \"kubernetes.io/projected/45ccdbed-282e-4205-a4f3-c535e41237ff-kube-api-access-6p5xn\") pod \"payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5\" (UID: \"45ccdbed-282e-4205-a4f3-c535e41237ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:28.593462 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.593431    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5xn\" (UniqueName: \"kubernetes.io/projected/45ccdbed-282e-4205-a4f3-c535e41237ff-kube-api-access-6p5xn\") pod \"payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5\" (UID: \"45ccdbed-282e-4205-a4f3-c535e41237ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:28.593832 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.593494    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccdbed-282e-4205-a4f3-c535e41237ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5\" (UID: \"45ccdbed-282e-4205-a4f3-c535e41237ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:28.594153 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.594133    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccdbed-282e-4205-a4f3-c535e41237ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5\" (UID: \"45ccdbed-282e-4205-a4f3-c535e41237ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:28.604543 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.604477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5xn\" (UniqueName: \"kubernetes.io/projected/45ccdbed-282e-4205-a4f3-c535e41237ff-kube-api-access-6p5xn\") pod \"payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5\" (UID: \"45ccdbed-282e-4205-a4f3-c535e41237ff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:28.637332 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.637295    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"]
Sep 16 19:05:28.640784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.640760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:28.645585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.645560    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d929c607-dockercfg-8w64n\""
Sep 16 19:05:28.652942 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.652917    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"]
Sep 16 19:05:28.694784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.694754    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdl5d\" (UniqueName: \"kubernetes.io/projected/ceae8c29-2f29-46ba-83b6-312428ed18bb-kube-api-access-hdl5d\") pod \"payload-processing-e2e-mt-d929c607-597bfd489-ct4l6\" (UID: \"ceae8c29-2f29-46ba-83b6-312428ed18bb\") " pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:28.694889 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.694799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceae8c29-2f29-46ba-83b6-312428ed18bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-d929c607-597bfd489-ct4l6\" (UID: \"ceae8c29-2f29-46ba-83b6-312428ed18bb\") " pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:28.792790 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.792758    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:28.795319 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.795294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceae8c29-2f29-46ba-83b6-312428ed18bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-d929c607-597bfd489-ct4l6\" (UID: \"ceae8c29-2f29-46ba-83b6-312428ed18bb\") " pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:28.795447 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.795364    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdl5d\" (UniqueName: \"kubernetes.io/projected/ceae8c29-2f29-46ba-83b6-312428ed18bb-kube-api-access-hdl5d\") pod \"payload-processing-e2e-mt-d929c607-597bfd489-ct4l6\" (UID: \"ceae8c29-2f29-46ba-83b6-312428ed18bb\") " pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:28.796061 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.796042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceae8c29-2f29-46ba-83b6-312428ed18bb-plugins-config-volume\") pod \"payload-processing-e2e-mt-d929c607-597bfd489-ct4l6\" (UID: \"ceae8c29-2f29-46ba-83b6-312428ed18bb\") " pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:28.811605 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.811584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdl5d\" (UniqueName: \"kubernetes.io/projected/ceae8c29-2f29-46ba-83b6-312428ed18bb-kube-api-access-hdl5d\") pod \"payload-processing-e2e-mt-d929c607-597bfd489-ct4l6\" (UID: \"ceae8c29-2f29-46ba-83b6-312428ed18bb\") " pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:28.840239 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.840186    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:28.952742 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:28.952652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:29.068387 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.068362    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"]
Sep 16 19:05:29.069410 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:05:29.069379    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ccdbed_282e_4205_a4f3_c535e41237ff.slice/crio-f70ffbb892a79e61356298dcf7c4cf112f1c81ec1286f1d6753bf0abb24f275a WatchSource:0}: Error finding container f70ffbb892a79e61356298dcf7c4cf112f1c81ec1286f1d6753bf0abb24f275a: Status 404 returned error can't find the container with id f70ffbb892a79e61356298dcf7c4cf112f1c81ec1286f1d6753bf0abb24f275a
Sep 16 19:05:29.228398 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.228357    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"]
Sep 16 19:05:29.232424 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:05:29.232397    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceae8c29_2f29_46ba_83b6_312428ed18bb.slice/crio-f46417cc39dec2e6b55d6c05a48c5e7526979af32d38d3eb65e0c6023da8e770 WatchSource:0}: Error finding container f46417cc39dec2e6b55d6c05a48c5e7526979af32d38d3eb65e0c6023da8e770: Status 404 returned error can't find the container with id f46417cc39dec2e6b55d6c05a48c5e7526979af32d38d3eb65e0c6023da8e770
Sep 16 19:05:29.432764 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.432724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6" event={"ID":"ceae8c29-2f29-46ba-83b6-312428ed18bb","Type":"ContainerStarted","Data":"6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1"}
Sep 16 19:05:29.432997 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.432793    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:29.432997 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.432816    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6" event={"ID":"ceae8c29-2f29-46ba-83b6-312428ed18bb","Type":"ContainerStarted","Data":"f46417cc39dec2e6b55d6c05a48c5e7526979af32d38d3eb65e0c6023da8e770"}
Sep 16 19:05:29.434464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.434436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5" event={"ID":"45ccdbed-282e-4205-a4f3-c535e41237ff","Type":"ContainerStarted","Data":"0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4"}
Sep 16 19:05:29.434592 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.434484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5" event={"ID":"45ccdbed-282e-4205-a4f3-c535e41237ff","Type":"ContainerStarted","Data":"f70ffbb892a79e61356298dcf7c4cf112f1c81ec1286f1d6753bf0abb24f275a"}
Sep 16 19:05:29.434667 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.434622    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:29.471702 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.471641    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6" podStartSLOduration=1.471624223 podStartE2EDuration="1.471624223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:29.47064273 +0000 UTC m=+1561.838698655" watchObservedRunningTime="2026-09-16 19:05:29.471624223 +0000 UTC m=+1561.839680124"
Sep 16 19:05:29.553997 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.553867    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5" podStartSLOduration=1.55385063 podStartE2EDuration="1.55385063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:29.553185968 +0000 UTC m=+1561.921241892" watchObservedRunningTime="2026-09-16 19:05:29.55385063 +0000 UTC m=+1561.921906530"
Sep 16 19:05:29.866777 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.866697    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:29.967633 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:29.967598    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:30.191595 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:30.191516    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:31.081847 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:31.081815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:31.399090 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:31.398992    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:31.980659 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:31.980624    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:32.907759 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:32.907725    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:33.810185 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:33.810155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:34.783823 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:34.783793    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:35.731630 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:35.731590    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:36.601837 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:36.601784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:36.919176 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:36.919096    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:40.441792 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:40.441763    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:05:40.442213 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:40.442022    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:05:40.699084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:40.699006    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:41.398184 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:41.398151    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:41.474263 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:41.474235    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:41.542619 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:41.542589    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:41.645173 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:41.645140    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:41.739183 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:41.739149    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:42.009566 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:42.009491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:42.127598 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:42.127559    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:42.743743 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:42.743676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:43.130015 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:43.129885    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:43.967680 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:43.967619    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:44.364124 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:44.364025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:45.507067 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:45.507022    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:46.319148 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:46.319098    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:47.119344 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:47.119297    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:49.126230 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:49.126193    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:49.621380 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:49.621347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:50.279436 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:50.279399    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:50.381363 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:50.381312    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:50.575049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:50.574950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:50.661609 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:50.661575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:51.127995 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:51.127942    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:51.523231 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:51.523202    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:52.047364 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:52.047331    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:54.108624 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:54.108591    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:54.726435 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:54.726393    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:54.815521 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:54.815491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:54.937359 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:54.937327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:55.807963 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:55.807931    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:56.482103 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:56.482071    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:56.612806 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:56.612759    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:57.608254 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:57.608203    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:58.405001 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:58.404957    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:59.446741 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:59.446711    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:59.543837 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:59.543802    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:59.635384 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:05:59.635345    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:00.103030 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.102996    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:00.204915 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.204872    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:00.315528 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.315491    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"]
Sep 16 19:06:00.318839 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.318821    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:00.323283 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.323256    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ca3f9a\""
Sep 16 19:06:00.323884 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.323865    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ca3f9a-dockercfg-9c6t7\""
Sep 16 19:06:00.331890 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.331869    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"]
Sep 16 19:06:00.497174 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.497131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsl8\" (UniqueName: \"kubernetes.io/projected/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-kube-api-access-tjsl8\") pod \"payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct\" (UID: \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:00.497564 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.497196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct\" (UID: \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:00.598107 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.598071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsl8\" (UniqueName: \"kubernetes.io/projected/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-kube-api-access-tjsl8\") pod \"payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct\" (UID: \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:00.598277 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.598131    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct\" (UID: \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:00.598811 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.598789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct\" (UID: \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:00.606685 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.606656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsl8\" (UniqueName: \"kubernetes.io/projected/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-kube-api-access-tjsl8\") pod \"payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct\" (UID: \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:00.629310 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.629283    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:00.917236 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:00.917211    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"]
Sep 16 19:06:00.919787 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:06:00.919752    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfd0e3e_01df_4ca5_bff3_d5a23c2c5d8f.slice/crio-af79fe0aa66aa3816340411c29d3a0442e998bbeb76f835663264f730780308f WatchSource:0}: Error finding container af79fe0aa66aa3816340411c29d3a0442e998bbeb76f835663264f730780308f: Status 404 returned error can't find the container with id af79fe0aa66aa3816340411c29d3a0442e998bbeb76f835663264f730780308f
Sep 16 19:06:01.593732 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:01.593693    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct" event={"ID":"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f","Type":"ContainerStarted","Data":"e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c"}
Sep 16 19:06:01.594142 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:01.593852    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:01.594142 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:01.593894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct" event={"ID":"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f","Type":"ContainerStarted","Data":"af79fe0aa66aa3816340411c29d3a0442e998bbeb76f835663264f730780308f"}
Sep 16 19:06:01.616581 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:01.616522    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct" podStartSLOduration=1.61650512 podStartE2EDuration="1.61650512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:01.615042588 +0000 UTC m=+1593.983098492" watchObservedRunningTime="2026-09-16 19:06:01.61650512 +0000 UTC m=+1593.984561046"
Sep 16 19:06:12.602046 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:12.602011    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:06:22.071426 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:22.071360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:23.493650 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:23.493613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:24.642683 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:24.642643    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:24.799269 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:24.799241    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:25.123340 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:25.123308    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:25.444884 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:25.444794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:25.626987 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:25.626945    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:26.023707 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:26.023676    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:26.405505 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:26.405423    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:27.285700 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:27.285666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:28.187393 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:28.187363    2559 scope.go:117] "RemoveContainer" containerID="a13a911032e58ea90e4e2418911326206e4b0246503aa1965701061c8f1aaef2"
Sep 16 19:06:28.223158 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:28.223131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:28.681171 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:28.681105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:28.910678 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:28.910646    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:29.018430 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:29.018398    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:29.140748 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:29.140696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:29.244860 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:29.244826    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:29.709113 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:29.709084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:31.022676 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:31.022649    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:31.994370 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:31.994315    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:33.045767 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:33.045734    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:34.090520 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:34.090486    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:34.188185 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:34.188153    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:34.948673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:34.948642    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:35.922709 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:35.922673    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:36.817393 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:36.817347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:36.929169 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:36.929136    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:37.103401 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.103323    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:37.506581 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.506551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:37.629157 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.629125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:37.770766 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.770678    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"]
Sep 16 19:06:37.774489 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.774465    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:37.779031 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.779013    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2dc9fc\""
Sep 16 19:06:37.779143 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.779059    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2dc9fc-dockercfg-5jptv\""
Sep 16 19:06:37.801991 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.799529    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"]
Sep 16 19:06:37.928897 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.928854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf\" (UID: \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:37.929098 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:37.929007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxxz\" (UniqueName: \"kubernetes.io/projected/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-kube-api-access-jkxxz\") pod \"payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf\" (UID: \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:38.029556 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.029469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxxz\" (UniqueName: \"kubernetes.io/projected/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-kube-api-access-jkxxz\") pod \"payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf\" (UID: \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:38.029901 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.029555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf\" (UID: \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:38.030258 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.030237    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf\" (UID: \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:38.044551 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.044519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxxz\" (UniqueName: \"kubernetes.io/projected/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-kube-api-access-jkxxz\") pod \"payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf\" (UID: \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:38.084117 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.084084    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:38.356143 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.356114    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"]
Sep 16 19:06:38.357660 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:06:38.357626    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba21cb9_148a_432d_b4c8_c2a8bf7de574.slice/crio-c5855fd3adda3561dfdeb6944911bbc220520e89d6f599bd4fd06d69a24c922c WatchSource:0}: Error finding container c5855fd3adda3561dfdeb6944911bbc220520e89d6f599bd4fd06d69a24c922c: Status 404 returned error can't find the container with id c5855fd3adda3561dfdeb6944911bbc220520e89d6f599bd4fd06d69a24c922c
Sep 16 19:06:38.777873 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.777839    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf" event={"ID":"7ba21cb9-148a-432d-b4c8-c2a8bf7de574","Type":"ContainerStarted","Data":"b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4"}
Sep 16 19:06:38.778096 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.777889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf" event={"ID":"7ba21cb9-148a-432d-b4c8-c2a8bf7de574","Type":"ContainerStarted","Data":"c5855fd3adda3561dfdeb6944911bbc220520e89d6f599bd4fd06d69a24c922c"}
Sep 16 19:06:38.778096 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.777966    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:38.811797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:38.811731    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf" podStartSLOduration=1.811716221 podStartE2EDuration="1.811716221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:38.810907586 +0000 UTC m=+1631.178963506" watchObservedRunningTime="2026-09-16 19:06:38.811716221 +0000 UTC m=+1631.179772175"
Sep 16 19:06:49.784960 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:49.784926    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:06:59.529954 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:06:59.529920    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:00.685077 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:00.685007    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:01.538407 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:01.538370    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:02.598088 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:02.598053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:03.403917 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:03.403872    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:04.125599 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:04.125565    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:04.271964 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:04.271932    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:05.254867 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:05.254830    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:06.389648 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:06.389612    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:09.380708 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:09.380664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:10.815587 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:10.815541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:11.227736 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:11.227691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:15.899673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:15.899627    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:18.134407 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:18.134369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:18.234263 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:18.234228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:20.218359 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:20.218320    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:28.206393 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:28.206355    2559 scope.go:117] "RemoveContainer" containerID="c2eae606a74a4992294938be0528b1f339883d0fd0749a1dd402b5b42b0b520e"
Sep 16 19:07:32.527190 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:32.527156    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:37.891794 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:37.891752    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:38.048750 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:38.048713    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:52.566995 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:52.566887    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:52.902125 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:52.902046    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:53.580431 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:53.580404    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:54.774452 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:54.774422    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:55.945114 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:55.945081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:56.826166 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:56.826136    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:57.917123 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:57.917082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:58.845004 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:58.844946    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:59.630263 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:07:59.630231    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:00.549714 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:00.549684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:03.520856 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:03.520818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:07.737512 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:07.737479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:07.784380 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:07.784349    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"]
Sep 16 19:08:07.784567 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:07.784544    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6" podUID="ceae8c29-2f29-46ba-83b6-312428ed18bb" containerName="payload-processing" containerID="cri-o://6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1" gracePeriod=30
Sep 16 19:08:07.797589 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:07.797554    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"]
Sep 16 19:08:07.797923 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:07.797876    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5" podUID="45ccdbed-282e-4205-a4f3-c535e41237ff" containerName="payload-pre-processing" containerID="cri-o://0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4" gracePeriod=30
Sep 16 19:08:07.828855 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:07.828800    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:07.930149 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:07.930120    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:08.024491 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.024462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:08.075392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.075361    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:08:08.091365 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.091333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:08.093191 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.093175    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:08:08.110901 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.110871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p5xn\" (UniqueName: \"kubernetes.io/projected/45ccdbed-282e-4205-a4f3-c535e41237ff-kube-api-access-6p5xn\") pod \"45ccdbed-282e-4205-a4f3-c535e41237ff\" (UID: \"45ccdbed-282e-4205-a4f3-c535e41237ff\") "
Sep 16 19:08:08.111105 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.110999    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdl5d\" (UniqueName: \"kubernetes.io/projected/ceae8c29-2f29-46ba-83b6-312428ed18bb-kube-api-access-hdl5d\") pod \"ceae8c29-2f29-46ba-83b6-312428ed18bb\" (UID: \"ceae8c29-2f29-46ba-83b6-312428ed18bb\") "
Sep 16 19:08:08.111105 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.111090    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceae8c29-2f29-46ba-83b6-312428ed18bb-plugins-config-volume\") pod \"ceae8c29-2f29-46ba-83b6-312428ed18bb\" (UID: \"ceae8c29-2f29-46ba-83b6-312428ed18bb\") "
Sep 16 19:08:08.111259 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.111128    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccdbed-282e-4205-a4f3-c535e41237ff-plugins-config-volume\") pod \"45ccdbed-282e-4205-a4f3-c535e41237ff\" (UID: \"45ccdbed-282e-4205-a4f3-c535e41237ff\") "
Sep 16 19:08:08.111588 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.111552    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceae8c29-2f29-46ba-83b6-312428ed18bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ceae8c29-2f29-46ba-83b6-312428ed18bb" (UID: "ceae8c29-2f29-46ba-83b6-312428ed18bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:08.111689 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.111614    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ccdbed-282e-4205-a4f3-c535e41237ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45ccdbed-282e-4205-a4f3-c535e41237ff" (UID: "45ccdbed-282e-4205-a4f3-c535e41237ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:08.113091 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.113070    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ccdbed-282e-4205-a4f3-c535e41237ff-kube-api-access-6p5xn" (OuterVolumeSpecName: "kube-api-access-6p5xn") pod "45ccdbed-282e-4205-a4f3-c535e41237ff" (UID: "45ccdbed-282e-4205-a4f3-c535e41237ff"). InnerVolumeSpecName "kube-api-access-6p5xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:08.113260 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.113245    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceae8c29-2f29-46ba-83b6-312428ed18bb-kube-api-access-hdl5d" (OuterVolumeSpecName: "kube-api-access-hdl5d") pod "ceae8c29-2f29-46ba-83b6-312428ed18bb" (UID: "ceae8c29-2f29-46ba-83b6-312428ed18bb"). InnerVolumeSpecName "kube-api-access-hdl5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:08.212682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.212652    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceae8c29-2f29-46ba-83b6-312428ed18bb-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.212682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.212679    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ccdbed-282e-4205-a4f3-c535e41237ff-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.212682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.212688    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5xn\" (UniqueName: \"kubernetes.io/projected/45ccdbed-282e-4205-a4f3-c535e41237ff-kube-api-access-6p5xn\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.212917 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.212697    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdl5d\" (UniqueName: \"kubernetes.io/projected/ceae8c29-2f29-46ba-83b6-312428ed18bb-kube-api-access-hdl5d\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:08:08.225578 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.225543    2559 generic.go:356] "Generic (PLEG): container finished" podID="ceae8c29-2f29-46ba-83b6-312428ed18bb" containerID="6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1" exitCode=0
Sep 16 19:08:08.225731 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.225581    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"
Sep 16 19:08:08.225731 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.225624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6" event={"ID":"ceae8c29-2f29-46ba-83b6-312428ed18bb","Type":"ContainerDied","Data":"6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1"}
Sep 16 19:08:08.225731 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.225685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6" event={"ID":"ceae8c29-2f29-46ba-83b6-312428ed18bb","Type":"ContainerDied","Data":"f46417cc39dec2e6b55d6c05a48c5e7526979af32d38d3eb65e0c6023da8e770"}
Sep 16 19:08:08.225731 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.225711    2559 scope.go:117] "RemoveContainer" containerID="6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1"
Sep 16 19:08:08.227421 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.227396    2559 generic.go:356] "Generic (PLEG): container finished" podID="45ccdbed-282e-4205-a4f3-c535e41237ff" containerID="0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4" exitCode=0
Sep 16 19:08:08.227550 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.227478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5" event={"ID":"45ccdbed-282e-4205-a4f3-c535e41237ff","Type":"ContainerDied","Data":"0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4"}
Sep 16 19:08:08.227550 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.227517    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5" event={"ID":"45ccdbed-282e-4205-a4f3-c535e41237ff","Type":"ContainerDied","Data":"f70ffbb892a79e61356298dcf7c4cf112f1c81ec1286f1d6753bf0abb24f275a"}
Sep 16 19:08:08.227550 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.227477    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"
Sep 16 19:08:08.236733 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.236711    2559 scope.go:117] "RemoveContainer" containerID="6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1"
Sep 16 19:08:08.238470 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:08:08.238446    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1\": container with ID starting with 6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1 not found: ID does not exist" containerID="6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1"
Sep 16 19:08:08.238590 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.238481    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1"} err="failed to get container status \"6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1\": rpc error: code = NotFound desc = could not find container \"6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1\": container with ID starting with 6da2f54c3125b77dea4935faca7c01f11a6340d0afad90f622f267cb0e50c1f1 not found: ID does not exist"
Sep 16 19:08:08.238590 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.238505    2559 scope.go:117] "RemoveContainer" containerID="0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4"
Sep 16 19:08:08.246830 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.246811    2559 scope.go:117] "RemoveContainer" containerID="0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4"
Sep 16 19:08:08.247127 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:08:08.247107    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4\": container with ID starting with 0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4 not found: ID does not exist" containerID="0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4"
Sep 16 19:08:08.247201 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.247131    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4"} err="failed to get container status \"0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4\": rpc error: code = NotFound desc = could not find container \"0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4\": container with ID starting with 0c320dc689f1c8d8958dd03b46d8128ddb50de9900f6ff4b0a47485b6b89f9e4 not found: ID does not exist"
Sep 16 19:08:08.256577 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.256556    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"]
Sep 16 19:08:08.258346 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.258298    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d929c607-644d456874-v6qv5"]
Sep 16 19:08:08.272382 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.272356    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"]
Sep 16 19:08:08.282259 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.282238    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d929c607-597bfd489-ct4l6"]
Sep 16 19:08:08.508754 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:08.508669    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:09.838499 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:09.838467    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:10.223324 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:10.223291    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ccdbed-282e-4205-a4f3-c535e41237ff" path="/var/lib/kubelet/pods/45ccdbed-282e-4205-a4f3-c535e41237ff/volumes"
Sep 16 19:08:10.223670 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:10.223657    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceae8c29-2f29-46ba-83b6-312428ed18bb" path="/var/lib/kubelet/pods/ceae8c29-2f29-46ba-83b6-312428ed18bb/volumes"
Sep 16 19:08:10.970924 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:10.970892    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:11.459759 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:11.459728    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:12.038243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:12.038211    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:12.963045 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:12.963011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:14.271617 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:14.271588    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:15.554248 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:15.554214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:16.566402 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:16.566366    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:41.886044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:41.886013    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:45.944456 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:45.944425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:53.481098 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:53.481036    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:53.620010 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:53.619963    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:54.934167 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:08:54.934133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:02.064920 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:02.064879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:02.533559 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:02.533525    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:02.641650 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:02.641616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:02.905179 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:02.905097    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:03.929258 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.929222    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"]
Sep 16 19:09:03.929631 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.929587    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ccdbed-282e-4205-a4f3-c535e41237ff" containerName="payload-pre-processing"
Sep 16 19:09:03.929631 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.929599    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ccdbed-282e-4205-a4f3-c535e41237ff" containerName="payload-pre-processing"
Sep 16 19:09:03.929631 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.929618    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceae8c29-2f29-46ba-83b6-312428ed18bb" containerName="payload-processing"
Sep 16 19:09:03.929631 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.929625    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceae8c29-2f29-46ba-83b6-312428ed18bb" containerName="payload-processing"
Sep 16 19:09:03.929766 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.929678    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ccdbed-282e-4205-a4f3-c535e41237ff" containerName="payload-pre-processing"
Sep 16 19:09:03.929766 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.929688    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceae8c29-2f29-46ba-83b6-312428ed18bb" containerName="payload-processing"
Sep 16 19:09:03.932637 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.932580    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:03.937912 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.937892    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c090c8\"/\"distinct-2-w4-c090c8-kserve-self-signed-certs\""
Sep 16 19:09:03.953279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:03.953248    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"]
Sep 16 19:09:04.007198 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.007162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tmp-dir\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.007198 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.007200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tls-certs\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.007459 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.007217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-dshm\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.007459 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.007232    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-model-cache\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.007459 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.007352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkzb\" (UniqueName: \"kubernetes.io/projected/376ea665-248b-42c0-b4e4-9c4864d5d3b1-kube-api-access-rzkzb\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.007459 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.007449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-home\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.026994 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.026911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:04.108156 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108121    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tmp-dir\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.108156 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108160    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tls-certs\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.108406 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108183    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-dshm\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.108406 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-model-cache\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.108406 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkzb\" (UniqueName: \"kubernetes.io/projected/376ea665-248b-42c0-b4e4-9c4864d5d3b1-kube-api-access-rzkzb\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.108406 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108330    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-home\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.108668 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tmp-dir\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.108733 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108683    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-home\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.108733 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.108664    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-model-cache\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.110535 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.110511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-dshm\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.110724 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.110707    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tls-certs\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.127126 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.127096    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkzb\" (UniqueName: \"kubernetes.io/projected/376ea665-248b-42c0-b4e4-9c4864d5d3b1-kube-api-access-rzkzb\") pod \"distinct-2-w4-c090c8-kserve-7d55c48d49-678m6\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") " pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.241457 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.241424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:04.398548 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.398519    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:04.547038 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.547008    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"]
Sep 16 19:09:04.549017 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:09:04.548985    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod376ea665_248b_42c0_b4e4_9c4864d5d3b1.slice/crio-4b5f7860a5c44c0bfdbb491dcf69f1be206f3e949a7c8479b0e4cdecc0220671 WatchSource:0}: Error finding container 4b5f7860a5c44c0bfdbb491dcf69f1be206f3e949a7c8479b0e4cdecc0220671: Status 404 returned error can't find the container with id 4b5f7860a5c44c0bfdbb491dcf69f1be206f3e949a7c8479b0e4cdecc0220671
Sep 16 19:09:04.569206 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:04.569179    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:05.435357 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:05.435309    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:05.503005 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:05.502956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6" event={"ID":"376ea665-248b-42c0-b4e4-9c4864d5d3b1","Type":"ContainerStarted","Data":"4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3"}
Sep 16 19:09:05.503161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:05.503017    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6" event={"ID":"376ea665-248b-42c0-b4e4-9c4864d5d3b1","Type":"ContainerStarted","Data":"4b5f7860a5c44c0bfdbb491dcf69f1be206f3e949a7c8479b0e4cdecc0220671"}
Sep 16 19:09:05.592112 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:05.592058    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6" podStartSLOduration=2.5920403260000002 podStartE2EDuration="2.592040326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:05.586728296 +0000 UTC m=+1777.954784197" watchObservedRunningTime="2026-09-16 19:09:05.592040326 +0000 UTC m=+1777.960096227"
Sep 16 19:09:05.592607 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:05.592584    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:07.175718 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:07.175666    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:14.242223 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:14.242184    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:14.242788 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:14.242236    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:14.254755 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:14.254729    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:14.559994 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:14.559894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:09:14.697345 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:14.697287    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:14.772056 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:14.772027    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:15.377667 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:15.377637    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:17.431445 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:17.431412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:23.801125 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:23.801090    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:23.973390 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:23.973359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:24.110349 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:24.110265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:24.212235 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:24.212203    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:28.100121 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:28.100095    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:09:28.103183 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:28.103157    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:09:34.683981 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:34.683934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:35.192994 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:35.192938    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:44.768352 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:44.768319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:55.895103 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:09:55.895067    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:02.956896 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:02.956851    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:03.169473 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:03.169443    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:03.352229 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:03.352152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:03.782628 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:03.782594    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:03.895227 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:03.895194    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:06.204838 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:06.204803    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:06.698620 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:06.698543    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:06.789170 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:06.789139    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:07.090012 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:07.089956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:08.250372 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:08.250334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:09.181531 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:09.181502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:10.737095 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:10.737060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:11.337631 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:11.337599    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:12.757323 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:12.757293    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:13.918262 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:13.918227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:14.034675 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.034638    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:14.606087 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.606045    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:14.729315 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.729285    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:14.830166 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.830132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:14.927945 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.927870    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"]
Sep 16 19:10:14.931251 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.931231    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:14.936234 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.936213    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a46230-dockercfg-4btpp\""
Sep 16 19:10:14.936383 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.936259    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a46230\""
Sep 16 19:10:14.946264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.946243    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"]
Sep 16 19:10:14.994164 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.994131    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"]
Sep 16 19:10:14.997187 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:14.997171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.016923 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.016897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"]
Sep 16 19:10:15.050645 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.050616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4252f\" (UniqueName: \"kubernetes.io/projected/49c14b22-6daa-49ea-aba0-42eb85276765-kube-api-access-4252f\") pod \"payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75\" (UID: \"49c14b22-6daa-49ea-aba0-42eb85276765\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:15.050821 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.050685    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c14b22-6daa-49ea-aba0-42eb85276765-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75\" (UID: \"49c14b22-6daa-49ea-aba0-42eb85276765\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:15.152198 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.152162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzqg\" (UniqueName: \"kubernetes.io/projected/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-kube-api-access-tlzqg\") pod \"payload-processing-e2e-api-a-a46230-5889689b59-8wx8q\" (UID: \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.152382 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.152227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4252f\" (UniqueName: \"kubernetes.io/projected/49c14b22-6daa-49ea-aba0-42eb85276765-kube-api-access-4252f\") pod \"payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75\" (UID: \"49c14b22-6daa-49ea-aba0-42eb85276765\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:15.152382 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.152262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a46230-5889689b59-8wx8q\" (UID: \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.152382 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.152306    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c14b22-6daa-49ea-aba0-42eb85276765-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75\" (UID: \"49c14b22-6daa-49ea-aba0-42eb85276765\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:15.153018 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.152995    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c14b22-6daa-49ea-aba0-42eb85276765-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75\" (UID: \"49c14b22-6daa-49ea-aba0-42eb85276765\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:15.167344 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.167316    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4252f\" (UniqueName: \"kubernetes.io/projected/49c14b22-6daa-49ea-aba0-42eb85276765-kube-api-access-4252f\") pod \"payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75\" (UID: \"49c14b22-6daa-49ea-aba0-42eb85276765\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:15.240565 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.240532    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:15.253784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.253759    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzqg\" (UniqueName: \"kubernetes.io/projected/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-kube-api-access-tlzqg\") pod \"payload-processing-e2e-api-a-a46230-5889689b59-8wx8q\" (UID: \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.253895 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.253830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a46230-5889689b59-8wx8q\" (UID: \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.254494 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.254472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a46230-5889689b59-8wx8q\" (UID: \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.263094 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.263073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzqg\" (UniqueName: \"kubernetes.io/projected/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-kube-api-access-tlzqg\") pod \"payload-processing-e2e-api-a-a46230-5889689b59-8wx8q\" (UID: \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\") " pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.307187 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.307083    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.514591 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.514564    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"]
Sep 16 19:10:15.517753 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:10:15.517719    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c14b22_6daa_49ea_aba0_42eb85276765.slice/crio-572a801ce8d3cbb81772f877e45c885db3a7bb147bedfe20ca40fc2af1861809 WatchSource:0}: Error finding container 572a801ce8d3cbb81772f877e45c885db3a7bb147bedfe20ca40fc2af1861809: Status 404 returned error can't find the container with id 572a801ce8d3cbb81772f877e45c885db3a7bb147bedfe20ca40fc2af1861809
Sep 16 19:10:15.584246 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.584219    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"]
Sep 16 19:10:15.584700 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:10:15.584676    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffcd93b_dc4e_4cf5_9a8d_8b7121532e0d.slice/crio-b7b0f4c39d45b95ecb7eb6efdf8d5973b75da9808c6cb92722cba89023eaf84e WatchSource:0}: Error finding container b7b0f4c39d45b95ecb7eb6efdf8d5973b75da9808c6cb92722cba89023eaf84e: Status 404 returned error can't find the container with id b7b0f4c39d45b95ecb7eb6efdf8d5973b75da9808c6cb92722cba89023eaf84e
Sep 16 19:10:15.841777 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.841688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q" event={"ID":"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d","Type":"ContainerStarted","Data":"0d40a160da9a0b3bc45eb3e13a5a66cfa6a93535d53dabdc3a0ab49a0b1b3e40"}
Sep 16 19:10:15.841777 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.841739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q" event={"ID":"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d","Type":"ContainerStarted","Data":"b7b0f4c39d45b95ecb7eb6efdf8d5973b75da9808c6cb92722cba89023eaf84e"}
Sep 16 19:10:15.842012 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.841789    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:15.843666 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.843636    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75" event={"ID":"49c14b22-6daa-49ea-aba0-42eb85276765","Type":"ContainerStarted","Data":"a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a"}
Sep 16 19:10:15.843804 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.843687    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:15.843804 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.843706    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75" event={"ID":"49c14b22-6daa-49ea-aba0-42eb85276765","Type":"ContainerStarted","Data":"572a801ce8d3cbb81772f877e45c885db3a7bb147bedfe20ca40fc2af1861809"}
Sep 16 19:10:15.866275 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.866207    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q" podStartSLOduration=1.866189113 podStartE2EDuration="1.866189113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:15.863537511 +0000 UTC m=+1848.231593431" watchObservedRunningTime="2026-09-16 19:10:15.866189113 +0000 UTC m=+1848.234245015"
Sep 16 19:10:15.885071 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.885020    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75" podStartSLOduration=1.8850063320000001 podStartE2EDuration="1.885006332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:15.882653268 +0000 UTC m=+1848.250709197" watchObservedRunningTime="2026-09-16 19:10:15.885006332 +0000 UTC m=+1848.253062236"
Sep 16 19:10:15.904940 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:15.904912    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:16.328881 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:16.328847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:17.398163 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:17.398129    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:18.370990 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:18.370943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:19.525695 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:19.525660    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:20.378646 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:20.378614    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:20.889865 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:20.889835    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"]
Sep 16 19:10:20.897359 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:20.897327    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"]
Sep 16 19:10:20.920662 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:20.920632    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"]
Sep 16 19:10:20.932141 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:20.932117    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"]
Sep 16 19:10:21.002102 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:21.002069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:21.428119 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:21.428082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:22.474243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:22.474204    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:23.406343 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:23.406314    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:24.877374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:24.877339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:25.775162 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:25.775124    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:26.501436 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:26.501402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:26.818343 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:26.818243    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:26.849587 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:26.849557    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:10:26.849754 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:26.849743    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:10:26.990384 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:26.990353    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:27.332229 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:27.332195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:27.969609 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:27.969578    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:28.340166 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:28.340081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:30.165206 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:30.165152    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:31.215686 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:31.215654    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:32.133193 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:32.133151    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:32.239236 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:32.239195    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:32.898344 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:32.898313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:34.000227 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.000196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:34.113237 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.113201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:34.315174 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.315094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:34.520964 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.520908    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:34.619455 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.619359    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:34.714410 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.714371    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"]
Sep 16 19:10:34.717532 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.717509    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:34.720721 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.720704    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b04e50\""
Sep 16 19:10:34.720838 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.720819    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b04e50-dockercfg-kv7xw\""
Sep 16 19:10:34.729846 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.729824    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"]
Sep 16 19:10:34.732240 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.732217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8da356-6f8f-4899-9787-a88f11c2bf1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt\" (UID: \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:34.732341 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.732291    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4f5v\" (UniqueName: \"kubernetes.io/projected/5c8da356-6f8f-4899-9787-a88f11c2bf1b-kube-api-access-r4f5v\") pod \"payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt\" (UID: \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:34.832836 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.832804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8da356-6f8f-4899-9787-a88f11c2bf1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt\" (UID: \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:34.833013 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.832924    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4f5v\" (UniqueName: \"kubernetes.io/projected/5c8da356-6f8f-4899-9787-a88f11c2bf1b-kube-api-access-r4f5v\") pod \"payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt\" (UID: \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:34.833427 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.833406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8da356-6f8f-4899-9787-a88f11c2bf1b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt\" (UID: \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:34.842026 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:34.842003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4f5v\" (UniqueName: \"kubernetes.io/projected/5c8da356-6f8f-4899-9787-a88f11c2bf1b-kube-api-access-r4f5v\") pod \"payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt\" (UID: \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\") " pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:35.028235 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.028206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:35.039840 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.039812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:35.303041 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.303020    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"]
Sep 16 19:10:35.305707 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:10:35.305662    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8da356_6f8f_4899_9787_a88f11c2bf1b.slice/crio-d051fc572041596d9e31386cca39c36877be4168d2418b1e03a1a69f1ce9c217 WatchSource:0}: Error finding container d051fc572041596d9e31386cca39c36877be4168d2418b1e03a1a69f1ce9c217: Status 404 returned error can't find the container with id d051fc572041596d9e31386cca39c36877be4168d2418b1e03a1a69f1ce9c217
Sep 16 19:10:35.909318 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.909258    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6" podUID="f4bc1e47-8865-4673-89b1-63751f3e2e98" containerName="main" containerID="cri-o://ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81" gracePeriod=45
Sep 16 19:10:35.916053 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.916016    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc" podUID="1d4437b7-681e-42f3-9839-f59e17b2b3ad" containerName="main" containerID="cri-o://3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6" gracePeriod=45
Sep 16 19:10:35.939273 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.939211    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg" podUID="f80a2441-8cef-4615-aa2b-ebf6eb34385e" containerName="main" containerID="cri-o://6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef" gracePeriod=45
Sep 16 19:10:35.946125 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.946093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt" event={"ID":"5c8da356-6f8f-4899-9787-a88f11c2bf1b","Type":"ContainerStarted","Data":"8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14"}
Sep 16 19:10:35.946246 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.946156    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt" event={"ID":"5c8da356-6f8f-4899-9787-a88f11c2bf1b","Type":"ContainerStarted","Data":"d051fc572041596d9e31386cca39c36877be4168d2418b1e03a1a69f1ce9c217"}
Sep 16 19:10:35.946246 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.946208    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:35.949693 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.949669    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n" podUID="ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" containerName="main" containerID="cri-o://5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338" gracePeriod=45
Sep 16 19:10:35.985742 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:35.985706    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc" podUID="1d4437b7-681e-42f3-9839-f59e17b2b3ad" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": dial tcp 10.132.0.52:8000: connect: connection refused"
Sep 16 19:10:36.239145 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.239113    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:10:36.241127 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.241098    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tls-certs\") pod \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") "
Sep 16 19:10:36.241582 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.241497    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-model-cache\") pod \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") "
Sep 16 19:10:36.241872 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.241779    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d4437b7-681e-42f3-9839-f59e17b2b3ad" (UID: "1d4437b7-681e-42f3-9839-f59e17b2b3ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.241872 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.241844    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-home\") pod \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") "
Sep 16 19:10:36.242106 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.242087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-home" (OuterVolumeSpecName: "home") pod "1d4437b7-681e-42f3-9839-f59e17b2b3ad" (UID: "1d4437b7-681e-42f3-9839-f59e17b2b3ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.242215 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.242199    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbnvk\" (UniqueName: \"kubernetes.io/projected/1d4437b7-681e-42f3-9839-f59e17b2b3ad-kube-api-access-mbnvk\") pod \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") "
Sep 16 19:10:36.242279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.242249    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-dshm\") pod \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") "
Sep 16 19:10:36.242650 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.242565    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tmp-dir\") pod \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\" (UID: \"1d4437b7-681e-42f3-9839-f59e17b2b3ad\") "
Sep 16 19:10:36.242884 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.242855    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d4437b7-681e-42f3-9839-f59e17b2b3ad" (UID: "1d4437b7-681e-42f3-9839-f59e17b2b3ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.244053 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.244033    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.244182 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.244169    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.244300 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.244288    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.246626 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.246596    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d4437b7-681e-42f3-9839-f59e17b2b3ad" (UID: "1d4437b7-681e-42f3-9839-f59e17b2b3ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:36.247242 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.246839    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4437b7-681e-42f3-9839-f59e17b2b3ad-kube-api-access-mbnvk" (OuterVolumeSpecName: "kube-api-access-mbnvk") pod "1d4437b7-681e-42f3-9839-f59e17b2b3ad" (UID: "1d4437b7-681e-42f3-9839-f59e17b2b3ad"). InnerVolumeSpecName "kube-api-access-mbnvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:36.247450 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.247418    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-dshm" (OuterVolumeSpecName: "dshm") pod "1d4437b7-681e-42f3-9839-f59e17b2b3ad" (UID: "1d4437b7-681e-42f3-9839-f59e17b2b3ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.258137 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.258065    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:10:36.262934 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.262902    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:10:36.266404 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.266384    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 19:10:36.278586 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.278518    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt" podStartSLOduration=2.278498002 podStartE2EDuration="2.278498002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:35.96742568 +0000 UTC m=+1868.335481608" watchObservedRunningTime="2026-09-16 19:10:36.278498002 +0000 UTC m=+1868.646553895"
Sep 16 19:10:36.303496 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.303462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:36.344688 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344629    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-home\") pod \"f4bc1e47-8865-4673-89b1-63751f3e2e98\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") "
Sep 16 19:10:36.344688 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344680    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-dshm\") pod \"f4bc1e47-8865-4673-89b1-63751f3e2e98\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344711    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tmp-dir\") pod \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344739    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tls-certs\") pod \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344764    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-tmp-dir\") pod \"f4bc1e47-8865-4673-89b1-63751f3e2e98\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344792    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-home\") pod \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344833    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-model-cache\") pod \"f4bc1e47-8865-4673-89b1-63751f3e2e98\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-dshm\") pod \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-model-cache\") pod \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344937    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-home" (OuterVolumeSpecName: "home") pod "f4bc1e47-8865-4673-89b1-63751f3e2e98" (UID: "f4bc1e47-8865-4673-89b1-63751f3e2e98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344953    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" (UID: "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.344956    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tmp-dir\") pod \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345020    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92w62\" (UniqueName: \"kubernetes.io/projected/f4bc1e47-8865-4673-89b1-63751f3e2e98-kube-api-access-92w62\") pod \"f4bc1e47-8865-4673-89b1-63751f3e2e98\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345062    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tls-certs\") pod \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345188    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f80a2441-8cef-4615-aa2b-ebf6eb34385e" (UID: "f80a2441-8cef-4615-aa2b-ebf6eb34385e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345191    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-home" (OuterVolumeSpecName: "home") pod "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" (UID: "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345407    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4bc1e47-8865-4673-89b1-63751f3e2e98" (UID: "f4bc1e47-8865-4673-89b1-63751f3e2e98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpftg\" (UniqueName: \"kubernetes.io/projected/f80a2441-8cef-4615-aa2b-ebf6eb34385e-kube-api-access-fpftg\") pod \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345642    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bc1e47-8865-4673-89b1-63751f3e2e98-tls-certs\") pod \"f4bc1e47-8865-4673-89b1-63751f3e2e98\" (UID: \"f4bc1e47-8865-4673-89b1-63751f3e2e98\") "
Sep 16 19:10:36.346200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345684    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-home\") pod \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") "
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345678    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" (UID: "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345714    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqgms\" (UniqueName: \"kubernetes.io/projected/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-kube-api-access-zqgms\") pod \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\" (UID: \"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e\") "
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345750    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-dshm\") pod \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") "
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.345779    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-model-cache\") pod \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\" (UID: \"f80a2441-8cef-4615-aa2b-ebf6eb34385e\") "
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346093    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346115    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbnvk\" (UniqueName: \"kubernetes.io/projected/1d4437b7-681e-42f3-9839-f59e17b2b3ad-kube-api-access-mbnvk\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346129    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d4437b7-681e-42f3-9839-f59e17b2b3ad-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346144    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d4437b7-681e-42f3-9839-f59e17b2b3ad-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346157    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346170    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346181    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346194    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346208    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346413    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f80a2441-8cef-4615-aa2b-ebf6eb34385e" (UID: "f80a2441-8cef-4615-aa2b-ebf6eb34385e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.346821    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4bc1e47-8865-4673-89b1-63751f3e2e98" (UID: "f4bc1e47-8865-4673-89b1-63751f3e2e98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.347129    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-home" (OuterVolumeSpecName: "home") pod "f80a2441-8cef-4615-aa2b-ebf6eb34385e" (UID: "f80a2441-8cef-4615-aa2b-ebf6eb34385e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.347673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.347481    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-dshm" (OuterVolumeSpecName: "dshm") pod "f4bc1e47-8865-4673-89b1-63751f3e2e98" (UID: "f4bc1e47-8865-4673-89b1-63751f3e2e98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.348929 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.348901    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bc1e47-8865-4673-89b1-63751f3e2e98-kube-api-access-92w62" (OuterVolumeSpecName: "kube-api-access-92w62") pod "f4bc1e47-8865-4673-89b1-63751f3e2e98" (UID: "f4bc1e47-8865-4673-89b1-63751f3e2e98"). InnerVolumeSpecName "kube-api-access-92w62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:36.349176 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.349152    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" (UID: "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:36.349597 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.349570    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f80a2441-8cef-4615-aa2b-ebf6eb34385e" (UID: "f80a2441-8cef-4615-aa2b-ebf6eb34385e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:36.349698 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.349604    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-dshm" (OuterVolumeSpecName: "dshm") pod "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" (UID: "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.350009 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.349965    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-dshm" (OuterVolumeSpecName: "dshm") pod "f80a2441-8cef-4615-aa2b-ebf6eb34385e" (UID: "f80a2441-8cef-4615-aa2b-ebf6eb34385e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:36.350126 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.350052    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80a2441-8cef-4615-aa2b-ebf6eb34385e-kube-api-access-fpftg" (OuterVolumeSpecName: "kube-api-access-fpftg") pod "f80a2441-8cef-4615-aa2b-ebf6eb34385e" (UID: "f80a2441-8cef-4615-aa2b-ebf6eb34385e"). InnerVolumeSpecName "kube-api-access-fpftg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:36.350243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.350225    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4bc1e47-8865-4673-89b1-63751f3e2e98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4bc1e47-8865-4673-89b1-63751f3e2e98" (UID: "f4bc1e47-8865-4673-89b1-63751f3e2e98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:36.350419 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.350394    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-kube-api-access-zqgms" (OuterVolumeSpecName: "kube-api-access-zqgms") pod "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" (UID: "ca1b65e7-3f6e-414b-b2e7-01be2b88da4e"). InnerVolumeSpecName "kube-api-access-zqgms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447419    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447445    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447454    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447463    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4bc1e47-8865-4673-89b1-63751f3e2e98-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447471    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447479    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92w62\" (UniqueName: \"kubernetes.io/projected/f4bc1e47-8865-4673-89b1-63751f3e2e98-kube-api-access-92w62\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447487    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80a2441-8cef-4615-aa2b-ebf6eb34385e-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447495    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpftg\" (UniqueName: \"kubernetes.io/projected/f80a2441-8cef-4615-aa2b-ebf6eb34385e-kube-api-access-fpftg\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447503    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4bc1e47-8865-4673-89b1-63751f3e2e98-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447870 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447512    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447870 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447519    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqgms\" (UniqueName: \"kubernetes.io/projected/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e-kube-api-access-zqgms\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.447870 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.447528    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80a2441-8cef-4615-aa2b-ebf6eb34385e-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:36.952790 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.952758    2559 generic.go:356] "Generic (PLEG): container finished" podID="f80a2441-8cef-4615-aa2b-ebf6eb34385e" containerID="6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef" exitCode=0
Sep 16 19:10:36.952919 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.952806    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"
Sep 16 19:10:36.952919 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.952845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg" event={"ID":"f80a2441-8cef-4615-aa2b-ebf6eb34385e","Type":"ContainerDied","Data":"6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef"}
Sep 16 19:10:36.952919 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.952896    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg" event={"ID":"f80a2441-8cef-4615-aa2b-ebf6eb34385e","Type":"ContainerDied","Data":"320b04dbafe9daa31872a891a91020fff945b52f92e1ddfba81a4fe8d6cbc113"}
Sep 16 19:10:36.952919 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.952917    2559 scope.go:117] "RemoveContainer" containerID="6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef"
Sep 16 19:10:36.954340 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.954315    2559 generic.go:356] "Generic (PLEG): container finished" podID="f4bc1e47-8865-4673-89b1-63751f3e2e98" containerID="ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81" exitCode=0
Sep 16 19:10:36.954457 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.954376    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"
Sep 16 19:10:36.954457 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.954418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6" event={"ID":"f4bc1e47-8865-4673-89b1-63751f3e2e98","Type":"ContainerDied","Data":"ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81"}
Sep 16 19:10:36.954570 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.954463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6" event={"ID":"f4bc1e47-8865-4673-89b1-63751f3e2e98","Type":"ContainerDied","Data":"a5572f888866219ac57acd40cbc8c9ee362dedacc560f5971656382129463576"}
Sep 16 19:10:36.955845 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.955827    2559 generic.go:356] "Generic (PLEG): container finished" podID="ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" containerID="5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338" exitCode=0
Sep 16 19:10:36.955925 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.955889    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"
Sep 16 19:10:36.956004 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.955930    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n" event={"ID":"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e","Type":"ContainerDied","Data":"5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338"}
Sep 16 19:10:36.956004 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.955958    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n" event={"ID":"ca1b65e7-3f6e-414b-b2e7-01be2b88da4e","Type":"ContainerDied","Data":"a76892b36bf3db36893d73e29913cc87a3e3338adec8c803fd64ab5b7ee45714"}
Sep 16 19:10:36.957541 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.957434    2559 generic.go:356] "Generic (PLEG): container finished" podID="1d4437b7-681e-42f3-9839-f59e17b2b3ad" containerID="3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6" exitCode=0
Sep 16 19:10:36.957541 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.957459    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc" event={"ID":"1d4437b7-681e-42f3-9839-f59e17b2b3ad","Type":"ContainerDied","Data":"3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6"}
Sep 16 19:10:36.957541 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.957479    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"
Sep 16 19:10:36.957541 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.957493    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc" event={"ID":"1d4437b7-681e-42f3-9839-f59e17b2b3ad","Type":"ContainerDied","Data":"b724354564d88a4b21c5a201360014aebc8798574fa5bc4997723daab90ce572"}
Sep 16 19:10:36.967167 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.966963    2559 scope.go:117] "RemoveContainer" containerID="6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef"
Sep 16 19:10:36.967784 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:10:36.967753    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef\": container with ID starting with 6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef not found: ID does not exist" containerID="6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef"
Sep 16 19:10:36.967871 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.967787    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef"} err="failed to get container status \"6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef\": rpc error: code = NotFound desc = could not find container \"6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef\": container with ID starting with 6935704ec4e4c76b686e4712f4f3f1fb1ea8fbbd344b0fdd427842659cab8eef not found: ID does not exist"
Sep 16 19:10:36.967871 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.967806    2559 scope.go:117] "RemoveContainer" containerID="ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81"
Sep 16 19:10:36.976474 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.976458    2559 scope.go:117] "RemoveContainer" containerID="ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81"
Sep 16 19:10:36.976726 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:10:36.976709    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81\": container with ID starting with ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81 not found: ID does not exist" containerID="ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81"
Sep 16 19:10:36.976784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.976731    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81"} err="failed to get container status \"ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81\": rpc error: code = NotFound desc = could not find container \"ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81\": container with ID starting with ec1a12db61ef174e44567a4f273a3c0f47396dae0223b5df079f32dd0ed0de81 not found: ID does not exist"
Sep 16 19:10:36.976784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.976745    2559 scope.go:117] "RemoveContainer" containerID="5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338"
Sep 16 19:10:36.984039 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.984021    2559 scope.go:117] "RemoveContainer" containerID="5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338"
Sep 16 19:10:36.984262 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:10:36.984245    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338\": container with ID starting with 5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338 not found: ID does not exist" containerID="5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338"
Sep 16 19:10:36.984320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.984267    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338"} err="failed to get container status \"5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338\": rpc error: code = NotFound desc = could not find container \"5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338\": container with ID starting with 5d0e1daa84ca8a3b9867b283736a01da49cefc9511384adee74f4967c0a95338 not found: ID does not exist"
Sep 16 19:10:36.984320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.984280    2559 scope.go:117] "RemoveContainer" containerID="3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6"
Sep 16 19:10:36.991366 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.991349    2559 scope.go:117] "RemoveContainer" containerID="3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6"
Sep 16 19:10:36.991648 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:10:36.991626    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6\": container with ID starting with 3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6 not found: ID does not exist" containerID="3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6"
Sep 16 19:10:36.991696 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:36.991659    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6"} err="failed to get container status \"3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6\": rpc error: code = NotFound desc = could not find container \"3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6\": container with ID starting with 3b777f9fdfa7740098f4ecfccd2b77f5c5732d98a94ca377db0cffe5474a7db6 not found: ID does not exist"
Sep 16 19:10:37.002156 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.002135    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"]
Sep 16 19:10:37.012622 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.012598    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/simulator-w0-b1857c-kserve-6f5fb67b4f-hrt8n"]
Sep 16 19:10:37.035947 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.035924    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"]
Sep 16 19:10:37.044524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.044498    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/premium-w0-b1857c-kserve-598c7946cb-jhhjg"]
Sep 16 19:10:37.069110 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.069076    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"]
Sep 16 19:10:37.075025 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.075005    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-w0-b1857c-kserve-5f69dc965f-djpzc"]
Sep 16 19:10:37.091847 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.091825    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"]
Sep 16 19:10:37.094423 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.094398    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1857c/distinct-2-w0-b1857c-kserve-579d65f7b8-bsxr6"]
Sep 16 19:10:37.415157 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:37.415125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:38.222841 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:38.222805    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4437b7-681e-42f3-9839-f59e17b2b3ad" path="/var/lib/kubelet/pods/1d4437b7-681e-42f3-9839-f59e17b2b3ad/volumes"
Sep 16 19:10:38.223235 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:38.223222    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" path="/var/lib/kubelet/pods/ca1b65e7-3f6e-414b-b2e7-01be2b88da4e/volumes"
Sep 16 19:10:38.223569 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:38.223558    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4bc1e47-8865-4673-89b1-63751f3e2e98" path="/var/lib/kubelet/pods/f4bc1e47-8865-4673-89b1-63751f3e2e98/volumes"
Sep 16 19:10:38.224203 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:38.224188    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f80a2441-8cef-4615-aa2b-ebf6eb34385e" path="/var/lib/kubelet/pods/f80a2441-8cef-4615-aa2b-ebf6eb34385e/volumes"
Sep 16 19:10:38.423168 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:38.423133    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:39.365380 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:39.365347    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:40.234123 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:40.234087    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:40.931280 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:40.931245    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:40.935616 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:40.935588    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"]
Sep 16 19:10:40.940210 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:40.940185    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1857c-29826420-jfzht"]
Sep 16 19:10:40.978653 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:40.978617    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"]
Sep 16 19:10:40.978859 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:40.978832    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm" podUID="ef377364-378f-45b9-be7c-b2fc66e57581" containerName="payload-processing" containerID="cri-o://906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088" gracePeriod=30
Sep 16 19:10:40.994379 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:40.994352    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"]
Sep 16 19:10:40.994583 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:40.994557    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2" podUID="4c3257b6-23d8-459b-9632-3c15f1efc165" containerName="payload-pre-processing" containerID="cri-o://d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648" gracePeriod=30
Sep 16 19:10:41.032223 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.032197    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:41.111923 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.111894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:41.209322 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.209289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:41.266489 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.266469    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 19:10:41.269312 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.269293    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 19:10:41.289372 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.289347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c3257b6-23d8-459b-9632-3c15f1efc165-plugins-config-volume\") pod \"4c3257b6-23d8-459b-9632-3c15f1efc165\" (UID: \"4c3257b6-23d8-459b-9632-3c15f1efc165\") "
Sep 16 19:10:41.289513 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.289426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef377364-378f-45b9-be7c-b2fc66e57581-plugins-config-volume\") pod \"ef377364-378f-45b9-be7c-b2fc66e57581\" (UID: \"ef377364-378f-45b9-be7c-b2fc66e57581\") "
Sep 16 19:10:41.289579 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.289545    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhm9f\" (UniqueName: \"kubernetes.io/projected/ef377364-378f-45b9-be7c-b2fc66e57581-kube-api-access-qhm9f\") pod \"ef377364-378f-45b9-be7c-b2fc66e57581\" (UID: \"ef377364-378f-45b9-be7c-b2fc66e57581\") "
Sep 16 19:10:41.289655 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.289603    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzbk2\" (UniqueName: \"kubernetes.io/projected/4c3257b6-23d8-459b-9632-3c15f1efc165-kube-api-access-wzbk2\") pod \"4c3257b6-23d8-459b-9632-3c15f1efc165\" (UID: \"4c3257b6-23d8-459b-9632-3c15f1efc165\") "
Sep 16 19:10:41.289773 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.289746    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3257b6-23d8-459b-9632-3c15f1efc165-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4c3257b6-23d8-459b-9632-3c15f1efc165" (UID: "4c3257b6-23d8-459b-9632-3c15f1efc165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:41.289904 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.289848    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef377364-378f-45b9-be7c-b2fc66e57581-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef377364-378f-45b9-be7c-b2fc66e57581" (UID: "ef377364-378f-45b9-be7c-b2fc66e57581"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:41.290238 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.290219    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c3257b6-23d8-459b-9632-3c15f1efc165-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.290340 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.290241    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef377364-378f-45b9-be7c-b2fc66e57581-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.292262 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.292241    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3257b6-23d8-459b-9632-3c15f1efc165-kube-api-access-wzbk2" (OuterVolumeSpecName: "kube-api-access-wzbk2") pod "4c3257b6-23d8-459b-9632-3c15f1efc165" (UID: "4c3257b6-23d8-459b-9632-3c15f1efc165"). InnerVolumeSpecName "kube-api-access-wzbk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:41.292343 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.292268    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef377364-378f-45b9-be7c-b2fc66e57581-kube-api-access-qhm9f" (OuterVolumeSpecName: "kube-api-access-qhm9f") pod "ef377364-378f-45b9-be7c-b2fc66e57581" (UID: "ef377364-378f-45b9-be7c-b2fc66e57581"). InnerVolumeSpecName "kube-api-access-qhm9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:41.391134 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.391105    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhm9f\" (UniqueName: \"kubernetes.io/projected/ef377364-378f-45b9-be7c-b2fc66e57581-kube-api-access-qhm9f\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.391272 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.391151    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbk2\" (UniqueName: \"kubernetes.io/projected/4c3257b6-23d8-459b-9632-3c15f1efc165-kube-api-access-wzbk2\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:41.866061 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.866029    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:41.977064 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.977030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:41.992941 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.992908    2559 generic.go:356] "Generic (PLEG): container finished" podID="4c3257b6-23d8-459b-9632-3c15f1efc165" containerID="d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648" exitCode=0
Sep 16 19:10:41.993090 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.992959    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"
Sep 16 19:10:41.993090 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.993011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2" event={"ID":"4c3257b6-23d8-459b-9632-3c15f1efc165","Type":"ContainerDied","Data":"d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648"}
Sep 16 19:10:41.993090 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.993087    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2" event={"ID":"4c3257b6-23d8-459b-9632-3c15f1efc165","Type":"ContainerDied","Data":"482009ae45215827c6112023bf0f30a205ec67c9401085ddc9eb17ae70395c73"}
Sep 16 19:10:41.993270 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.993105    2559 scope.go:117] "RemoveContainer" containerID="d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648"
Sep 16 19:10:41.994508 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.994487    2559 generic.go:356] "Generic (PLEG): container finished" podID="ef377364-378f-45b9-be7c-b2fc66e57581" containerID="906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088" exitCode=0
Sep 16 19:10:41.994607 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.994536    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"
Sep 16 19:10:41.994607 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.994555    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm" event={"ID":"ef377364-378f-45b9-be7c-b2fc66e57581","Type":"ContainerDied","Data":"906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088"}
Sep 16 19:10:41.994607 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:41.994594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm" event={"ID":"ef377364-378f-45b9-be7c-b2fc66e57581","Type":"ContainerDied","Data":"8fd1378b778b12385dec8780c52ae95236eff4a2f057b7a315b5d6b56ce6a288"}
Sep 16 19:10:42.010448 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.010426    2559 scope.go:117] "RemoveContainer" containerID="d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648"
Sep 16 19:10:42.010727 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:10:42.010707    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648\": container with ID starting with d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648 not found: ID does not exist" containerID="d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648"
Sep 16 19:10:42.010799 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.010734    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648"} err="failed to get container status \"d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648\": rpc error: code = NotFound desc = could not find container \"d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648\": container with ID starting with d16112091e4b723b0c9858b67f61042f8e171d1f3179b99f123db41e0bf6d648 not found: ID does not exist"
Sep 16 19:10:42.010799 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.010751    2559 scope.go:117] "RemoveContainer" containerID="906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088"
Sep 16 19:10:42.018646 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.018630    2559 scope.go:117] "RemoveContainer" containerID="906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088"
Sep 16 19:10:42.018935 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:10:42.018907    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088\": container with ID starting with 906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088 not found: ID does not exist" containerID="906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088"
Sep 16 19:10:42.018935 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.018928    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088"} err="failed to get container status \"906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088\": rpc error: code = NotFound desc = could not find container \"906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088\": container with ID starting with 906a4a2e96e77d93d31fafd559c6375960e6153cda23bd006ebca0e747576088 not found: ID does not exist"
Sep 16 19:10:42.067797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.067766    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"]
Sep 16 19:10:42.072031 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.072003    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b1857c-56898898c-79pbm"]
Sep 16 19:10:42.091865 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.091842    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"]
Sep 16 19:10:42.102292 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.102271    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2"]
Sep 16 19:10:42.222941 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.222908    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a41f00-6bcf-4445-864c-d5f147f682f4" path="/var/lib/kubelet/pods/04a41f00-6bcf-4445-864c-d5f147f682f4/volumes"
Sep 16 19:10:42.223272 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.223258    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c3257b6-23d8-459b-9632-3c15f1efc165" path="/var/lib/kubelet/pods/4c3257b6-23d8-459b-9632-3c15f1efc165/volumes"
Sep 16 19:10:42.223604 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.223580    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef377364-378f-45b9-be7c-b2fc66e57581" path="/var/lib/kubelet/pods/ef377364-378f-45b9-be7c-b2fc66e57581/volumes"
Sep 16 19:10:42.263550 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:42.263512    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1857c-9b6667bfd-vn7c2" podUID="4c3257b6-23d8-459b-9632-3c15f1efc165" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.30:9004: i/o timeout"
Sep 16 19:10:43.023685 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:43.023656    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:43.949177 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:43.949143    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:45.162337 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:45.162299    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:45.978572 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:45.978539    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:46.916311 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:46.916277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:46.960355 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:46.960324    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:10:47.955456 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:47.955420    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:48.617115 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:48.617082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:48.715948 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:48.715916    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:48.816611 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:48.816575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:48.919279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:48.919200    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:49.175167 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:49.175084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:50.887020 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:50.886929    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:51.418961 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:51.418926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:52.492654 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:52.492617    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:53.212724 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:53.212688    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:53.432697 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:53.432668    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:54.009177 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009136    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"]
Sep 16 19:10:54.009561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009512    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" containerName="main"
Sep 16 19:10:54.009561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009524    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" containerName="main"
Sep 16 19:10:54.009561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009533    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f80a2441-8cef-4615-aa2b-ebf6eb34385e" containerName="main"
Sep 16 19:10:54.009561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009538    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f80a2441-8cef-4615-aa2b-ebf6eb34385e" containerName="main"
Sep 16 19:10:54.009561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009546    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c3257b6-23d8-459b-9632-3c15f1efc165" containerName="payload-pre-processing"
Sep 16 19:10:54.009561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009552    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c3257b6-23d8-459b-9632-3c15f1efc165" containerName="payload-pre-processing"
Sep 16 19:10:54.009561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009560    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4bc1e47-8865-4673-89b1-63751f3e2e98" containerName="main"
Sep 16 19:10:54.009561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009565    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4bc1e47-8865-4673-89b1-63751f3e2e98" containerName="main"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009574    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4437b7-681e-42f3-9839-f59e17b2b3ad" containerName="main"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009579    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4437b7-681e-42f3-9839-f59e17b2b3ad" containerName="main"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009589    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef377364-378f-45b9-be7c-b2fc66e57581" containerName="payload-processing"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009594    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef377364-378f-45b9-be7c-b2fc66e57581" containerName="payload-processing"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009650    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c3257b6-23d8-459b-9632-3c15f1efc165" containerName="payload-pre-processing"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009658    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1b65e7-3f6e-414b-b2e7-01be2b88da4e" containerName="main"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009662    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f80a2441-8cef-4615-aa2b-ebf6eb34385e" containerName="main"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009669    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4437b7-681e-42f3-9839-f59e17b2b3ad" containerName="main"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009673    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef377364-378f-45b9-be7c-b2fc66e57581" containerName="payload-processing"
Sep 16 19:10:54.009807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.009679    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4bc1e47-8865-4673-89b1-63751f3e2e98" containerName="main"
Sep 16 19:10:54.015533 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.015515    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"
Sep 16 19:10:54.018365 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.018347    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:10:54.018625 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.018609    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:10:54.033283 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.033252    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"]
Sep 16 19:10:54.095305 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.095267    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35aedbca-6fca-44c6-b018-6910d9ab6447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a46230-df5gn\" (UID: \"35aedbca-6fca-44c6-b018-6910d9ab6447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"
Sep 16 19:10:54.196415 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.196377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35aedbca-6fca-44c6-b018-6910d9ab6447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a46230-df5gn\" (UID: \"35aedbca-6fca-44c6-b018-6910d9ab6447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"
Sep 16 19:10:54.197026 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.197007    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35aedbca-6fca-44c6-b018-6910d9ab6447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a46230-df5gn\" (UID: \"35aedbca-6fca-44c6-b018-6910d9ab6447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"
Sep 16 19:10:54.325054 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.324962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"
Sep 16 19:10:54.475737 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.475704    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:54.611904 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:54.611855    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"]
Sep 16 19:10:54.613795 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:10:54.613765    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35aedbca_6fca_44c6_b018_6910d9ab6447.slice/crio-2ac6c6b831335b2e133e6f3fa2aa478dbcd8063965f3074a1f09336c08f19d2c WatchSource:0}: Error finding container 2ac6c6b831335b2e133e6f3fa2aa478dbcd8063965f3074a1f09336c08f19d2c: Status 404 returned error can't find the container with id 2ac6c6b831335b2e133e6f3fa2aa478dbcd8063965f3074a1f09336c08f19d2c
Sep 16 19:10:55.062855 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:55.062812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn" event={"ID":"35aedbca-6fca-44c6-b018-6910d9ab6447","Type":"ContainerStarted","Data":"245ab00e5db83d2239a028f18e95ac6f505881dada2b95bfd25f60494e827ca9"}
Sep 16 19:10:55.063209 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:55.062874    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn" event={"ID":"35aedbca-6fca-44c6-b018-6910d9ab6447","Type":"ContainerStarted","Data":"2ac6c6b831335b2e133e6f3fa2aa478dbcd8063965f3074a1f09336c08f19d2c"}
Sep 16 19:10:55.086088 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:55.086027    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn" podStartSLOduration=2.086008086 podStartE2EDuration="2.086008086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:55.08482499 +0000 UTC m=+1887.452880892" watchObservedRunningTime="2026-09-16 19:10:55.086008086 +0000 UTC m=+1887.454063986"
Sep 16 19:10:55.572355 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:55.572321    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:56.069565 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:56.069528    2559 generic.go:356] "Generic (PLEG): container finished" podID="35aedbca-6fca-44c6-b018-6910d9ab6447" containerID="245ab00e5db83d2239a028f18e95ac6f505881dada2b95bfd25f60494e827ca9" exitCode=0
Sep 16 19:10:56.069937 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:56.069602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn" event={"ID":"35aedbca-6fca-44c6-b018-6910d9ab6447","Type":"ContainerDied","Data":"245ab00e5db83d2239a028f18e95ac6f505881dada2b95bfd25f60494e827ca9"}
Sep 16 19:10:56.495535 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:56.495503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:57.209574 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:57.209547    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"
Sep 16 19:10:57.322506 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:57.322466    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35aedbca-6fca-44c6-b018-6910d9ab6447-maas-api-service-ca\") pod \"35aedbca-6fca-44c6-b018-6910d9ab6447\" (UID: \"35aedbca-6fca-44c6-b018-6910d9ab6447\") "
Sep 16 19:10:57.322860 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:57.322839    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35aedbca-6fca-44c6-b018-6910d9ab6447-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35aedbca-6fca-44c6-b018-6910d9ab6447" (UID: "35aedbca-6fca-44c6-b018-6910d9ab6447"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:10:57.423943 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:57.423844    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35aedbca-6fca-44c6-b018-6910d9ab6447-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:10:57.944830 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:57.944795    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:58.036155 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:58.036122    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:58.080261 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:58.080231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn" event={"ID":"35aedbca-6fca-44c6-b018-6910d9ab6447","Type":"ContainerDied","Data":"2ac6c6b831335b2e133e6f3fa2aa478dbcd8063965f3074a1f09336c08f19d2c"}
Sep 16 19:10:58.080397 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:58.080281    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ac6c6b831335b2e133e6f3fa2aa478dbcd8063965f3074a1f09336c08f19d2c"
Sep 16 19:10:58.080397 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:58.080243    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"
Sep 16 19:10:58.381778 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:58.381745    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:58.483710 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:58.483671    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:59.169379 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:10:59.169340    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:00.101228 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:00.101191    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:01.063711 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:01.063678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:01.563630 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:01.563596    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:02.022682 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:02.022652    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:04.171428 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:04.171392    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:08.906448 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:08.906415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:09.973334 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:09.973290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:10.009831 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.009801    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"]
Sep 16 19:11:10.010055 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.010034    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q" podUID="4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d" containerName="payload-processing" containerID="cri-o://0d40a160da9a0b3bc45eb3e13a5a66cfa6a93535d53dabdc3a0ab49a0b1b3e40" gracePeriod=30
Sep 16 19:11:10.029915 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.029891    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"]
Sep 16 19:11:10.030159 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.030114    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75" podUID="49c14b22-6daa-49ea-aba0-42eb85276765" containerName="payload-pre-processing" containerID="cri-o://a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a" gracePeriod=30
Sep 16 19:11:10.066784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.066749    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:10.136674 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.136634    2559 generic.go:356] "Generic (PLEG): container finished" podID="4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d" containerID="0d40a160da9a0b3bc45eb3e13a5a66cfa6a93535d53dabdc3a0ab49a0b1b3e40" exitCode=0
Sep 16 19:11:10.136727 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.136704    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q" event={"ID":"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d","Type":"ContainerDied","Data":"0d40a160da9a0b3bc45eb3e13a5a66cfa6a93535d53dabdc3a0ab49a0b1b3e40"}
Sep 16 19:11:10.173778 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.173747    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:10.299727 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.299695    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:11:10.302697 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.302676    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:11:10.338855 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.338823    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"]
Sep 16 19:11:10.342331 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.342292    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a46230-df5gn"]
Sep 16 19:11:10.437564 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.437533    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4252f\" (UniqueName: \"kubernetes.io/projected/49c14b22-6daa-49ea-aba0-42eb85276765-kube-api-access-4252f\") pod \"49c14b22-6daa-49ea-aba0-42eb85276765\" (UID: \"49c14b22-6daa-49ea-aba0-42eb85276765\") "
Sep 16 19:11:10.437745 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.437595    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c14b22-6daa-49ea-aba0-42eb85276765-plugins-config-volume\") pod \"49c14b22-6daa-49ea-aba0-42eb85276765\" (UID: \"49c14b22-6daa-49ea-aba0-42eb85276765\") "
Sep 16 19:11:10.437745 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.437648    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-plugins-config-volume\") pod \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\" (UID: \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\") "
Sep 16 19:11:10.437745 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.437678    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlzqg\" (UniqueName: \"kubernetes.io/projected/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-kube-api-access-tlzqg\") pod \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\" (UID: \"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d\") "
Sep 16 19:11:10.438091 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.438066    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c14b22-6daa-49ea-aba0-42eb85276765-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49c14b22-6daa-49ea-aba0-42eb85276765" (UID: "49c14b22-6daa-49ea-aba0-42eb85276765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:10.438171 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.438066    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d" (UID: "4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:10.439900 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.439876    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-kube-api-access-tlzqg" (OuterVolumeSpecName: "kube-api-access-tlzqg") pod "4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d" (UID: "4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d"). InnerVolumeSpecName "kube-api-access-tlzqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:10.440003 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.439903    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c14b22-6daa-49ea-aba0-42eb85276765-kube-api-access-4252f" (OuterVolumeSpecName: "kube-api-access-4252f") pod "49c14b22-6daa-49ea-aba0-42eb85276765" (UID: "49c14b22-6daa-49ea-aba0-42eb85276765"). InnerVolumeSpecName "kube-api-access-4252f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:10.538577 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.538500    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4252f\" (UniqueName: \"kubernetes.io/projected/49c14b22-6daa-49ea-aba0-42eb85276765-kube-api-access-4252f\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.538577 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.538530    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c14b22-6daa-49ea-aba0-42eb85276765-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.538577 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.538541    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:10.538577 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:10.538552    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzqg\" (UniqueName: \"kubernetes.io/projected/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d-kube-api-access-tlzqg\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:11.035083 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.035050    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:11.131276 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.131243    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:11.142575 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.142543    2559 generic.go:356] "Generic (PLEG): container finished" podID="49c14b22-6daa-49ea-aba0-42eb85276765" containerID="a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a" exitCode=0
Sep 16 19:11:11.142724 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.142591    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"
Sep 16 19:11:11.142724 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.142629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75" event={"ID":"49c14b22-6daa-49ea-aba0-42eb85276765","Type":"ContainerDied","Data":"a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a"}
Sep 16 19:11:11.142724 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.142688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75" event={"ID":"49c14b22-6daa-49ea-aba0-42eb85276765","Type":"ContainerDied","Data":"572a801ce8d3cbb81772f877e45c885db3a7bb147bedfe20ca40fc2af1861809"}
Sep 16 19:11:11.142724 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.142711    2559 scope.go:117] "RemoveContainer" containerID="a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a"
Sep 16 19:11:11.144263 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.144246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q" event={"ID":"4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d","Type":"ContainerDied","Data":"b7b0f4c39d45b95ecb7eb6efdf8d5973b75da9808c6cb92722cba89023eaf84e"}
Sep 16 19:11:11.144370 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.144248    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"
Sep 16 19:11:11.151408 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.151385    2559 scope.go:117] "RemoveContainer" containerID="a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a"
Sep 16 19:11:11.151727 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:11:11.151695    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a\": container with ID starting with a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a not found: ID does not exist" containerID="a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a"
Sep 16 19:11:11.151956 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.151735    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a"} err="failed to get container status \"a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a\": rpc error: code = NotFound desc = could not find container \"a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a\": container with ID starting with a9343795485637750fc675d9d89612a56eebf7f6508516054d2e17c66b00dc9a not found: ID does not exist"
Sep 16 19:11:11.151956 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.151752    2559 scope.go:117] "RemoveContainer" containerID="0d40a160da9a0b3bc45eb3e13a5a66cfa6a93535d53dabdc3a0ab49a0b1b3e40"
Sep 16 19:11:11.175936 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.175910    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"]
Sep 16 19:11:11.179634 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.179611    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a46230-fc6974f94-z5r75"]
Sep 16 19:11:11.192728 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.192706    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"]
Sep 16 19:11:11.195879 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:11.195853    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a46230-5889689b59-8wx8q"]
Sep 16 19:11:12.025516 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:12.025486    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:12.221942 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:12.221907    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35aedbca-6fca-44c6-b018-6910d9ab6447" path="/var/lib/kubelet/pods/35aedbca-6fca-44c6-b018-6910d9ab6447/volumes"
Sep 16 19:11:12.222470 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:12.222449    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49c14b22-6daa-49ea-aba0-42eb85276765" path="/var/lib/kubelet/pods/49c14b22-6daa-49ea-aba0-42eb85276765/volumes"
Sep 16 19:11:12.222917 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:12.222901    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d" path="/var/lib/kubelet/pods/4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d/volumes"
Sep 16 19:11:13.034852 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:13.034812    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:13.982655 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:13.982617    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:14.795396 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:14.795361    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:15.668520 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:15.668487    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:16.614014 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:16.613963    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:18.607473 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:18.607429    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:18.699956 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:18.699922    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:18.910905 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:18.910802    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:19.228618 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.228589    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"]
Sep 16 19:11:19.228910 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.228898    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35aedbca-6fca-44c6-b018-6910d9ab6447" containerName="revoke-keys"
Sep 16 19:11:19.228955 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.228911    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="35aedbca-6fca-44c6-b018-6910d9ab6447" containerName="revoke-keys"
Sep 16 19:11:19.228955 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.228933    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49c14b22-6daa-49ea-aba0-42eb85276765" containerName="payload-pre-processing"
Sep 16 19:11:19.228955 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.228939    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="49c14b22-6daa-49ea-aba0-42eb85276765" containerName="payload-pre-processing"
Sep 16 19:11:19.228955 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.228947    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d" containerName="payload-processing"
Sep 16 19:11:19.228955 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.228954    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d" containerName="payload-processing"
Sep 16 19:11:19.229140 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.229023    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ffcd93b-dc4e-4cf5-9a8d-8b7121532e0d" containerName="payload-processing"
Sep 16 19:11:19.229140 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.229034    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="49c14b22-6daa-49ea-aba0-42eb85276765" containerName="payload-pre-processing"
Sep 16 19:11:19.229140 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.229042    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="35aedbca-6fca-44c6-b018-6910d9ab6447" containerName="revoke-keys"
Sep 16 19:11:19.233208 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.233190    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"
Sep 16 19:11:19.236925 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.236907    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:11:19.237053 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.236907    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:11:19.240772 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.240746    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"]
Sep 16 19:11:19.323503 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.323468    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bd9a22-41de-4f99-817f-da2731c123e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv\" (UID: \"55bd9a22-41de-4f99-817f-da2731c123e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"
Sep 16 19:11:19.423935 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.423896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bd9a22-41de-4f99-817f-da2731c123e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv\" (UID: \"55bd9a22-41de-4f99-817f-da2731c123e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"
Sep 16 19:11:19.424553 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.424532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bd9a22-41de-4f99-817f-da2731c123e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv\" (UID: \"55bd9a22-41de-4f99-817f-da2731c123e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"
Sep 16 19:11:19.514557 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.514462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:19.545042 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.545019    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"
Sep 16 19:11:19.828291 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:11:19.828263    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bd9a22_41de_4f99_817f_da2731c123e4.slice/crio-ee23ab23ee650dc69023e23e028f7e5d452a257e4b3bd44dea8091e7d17f1ae1 WatchSource:0}: Error finding container ee23ab23ee650dc69023e23e028f7e5d452a257e4b3bd44dea8091e7d17f1ae1: Status 404 returned error can't find the container with id ee23ab23ee650dc69023e23e028f7e5d452a257e4b3bd44dea8091e7d17f1ae1
Sep 16 19:11:19.832477 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:19.832454    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"]
Sep 16 19:11:20.190271 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:20.190175    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv" event={"ID":"55bd9a22-41de-4f99-817f-da2731c123e4","Type":"ContainerStarted","Data":"79977d88aeb9ffb40c93e5382aa7c986f5780afb4a1f330e055cf9a6c534ed73"}
Sep 16 19:11:20.190271 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:20.190262    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv" event={"ID":"55bd9a22-41de-4f99-817f-da2731c123e4","Type":"ContainerStarted","Data":"ee23ab23ee650dc69023e23e028f7e5d452a257e4b3bd44dea8091e7d17f1ae1"}
Sep 16 19:11:20.212066 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:20.212010    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv" podStartSLOduration=1.2119931290000001 podStartE2EDuration="1.211993129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:20.21128118 +0000 UTC m=+1912.579337102" watchObservedRunningTime="2026-09-16 19:11:20.211993129 +0000 UTC m=+1912.580049029"
Sep 16 19:11:20.670281 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:20.670244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:21.196758 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:21.196724    2559 generic.go:356] "Generic (PLEG): container finished" podID="55bd9a22-41de-4f99-817f-da2731c123e4" containerID="79977d88aeb9ffb40c93e5382aa7c986f5780afb4a1f330e055cf9a6c534ed73" exitCode=0
Sep 16 19:11:21.197221 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:21.196801    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv" event={"ID":"55bd9a22-41de-4f99-817f-da2731c123e4","Type":"ContainerDied","Data":"79977d88aeb9ffb40c93e5382aa7c986f5780afb4a1f330e055cf9a6c534ed73"}
Sep 16 19:11:21.617343 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:21.617279    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:22.334531 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:22.334508    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"
Sep 16 19:11:22.378779 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:22.378746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:22.451238 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:22.451205    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bd9a22-41de-4f99-817f-da2731c123e4-maas-api-service-ca\") pod \"55bd9a22-41de-4f99-817f-da2731c123e4\" (UID: \"55bd9a22-41de-4f99-817f-da2731c123e4\") "
Sep 16 19:11:22.451610 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:22.451586    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bd9a22-41de-4f99-817f-da2731c123e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55bd9a22-41de-4f99-817f-da2731c123e4" (UID: "55bd9a22-41de-4f99-817f-da2731c123e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:22.552105 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:22.552073    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bd9a22-41de-4f99-817f-da2731c123e4-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:23.208226 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:23.208196    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"
Sep 16 19:11:23.208400 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:23.208240    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv" event={"ID":"55bd9a22-41de-4f99-817f-da2731c123e4","Type":"ContainerDied","Data":"ee23ab23ee650dc69023e23e028f7e5d452a257e4b3bd44dea8091e7d17f1ae1"}
Sep 16 19:11:23.208400 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:23.208290    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee23ab23ee650dc69023e23e028f7e5d452a257e4b3bd44dea8091e7d17f1ae1"
Sep 16 19:11:23.430983 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:23.430953    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:24.211020 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:24.210984    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:24.982798 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:24.982761    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:25.833359 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:25.833324    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:28.257694 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:28.257659    2559 scope.go:117] "RemoveContainer" containerID="ebbfdf86b74f2c7ae1a185fecacf3eba145436141b06c9ef1db5ba851f978a48"
Sep 16 19:11:34.900358 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:34.900328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:34.930257 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:34.930227    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"]
Sep 16 19:11:34.930461 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:34.930436    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt" podUID="5c8da356-6f8f-4899-9787-a88f11c2bf1b" containerName="payload-processing" containerID="cri-o://8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14" gracePeriod=30
Sep 16 19:11:34.993385 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:34.993355    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:35.104162 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.104132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:35.178292 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.178269    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:11:35.271602 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.267453    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8da356-6f8f-4899-9787-a88f11c2bf1b-plugins-config-volume\") pod \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\" (UID: \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\") "
Sep 16 19:11:35.271602 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.267538    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4f5v\" (UniqueName: \"kubernetes.io/projected/5c8da356-6f8f-4899-9787-a88f11c2bf1b-kube-api-access-r4f5v\") pod \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\" (UID: \"5c8da356-6f8f-4899-9787-a88f11c2bf1b\") "
Sep 16 19:11:35.272587 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.272553    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8da356-6f8f-4899-9787-a88f11c2bf1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c8da356-6f8f-4899-9787-a88f11c2bf1b" (UID: "5c8da356-6f8f-4899-9787-a88f11c2bf1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:35.272852 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.272829    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:35.273815 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.273778    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8da356-6f8f-4899-9787-a88f11c2bf1b-kube-api-access-r4f5v" (OuterVolumeSpecName: "kube-api-access-r4f5v") pod "5c8da356-6f8f-4899-9787-a88f11c2bf1b" (UID: "5c8da356-6f8f-4899-9787-a88f11c2bf1b"). InnerVolumeSpecName "kube-api-access-r4f5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:35.276389 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.276361    2559 generic.go:356] "Generic (PLEG): container finished" podID="5c8da356-6f8f-4899-9787-a88f11c2bf1b" containerID="8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14" exitCode=0
Sep 16 19:11:35.276483 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.276412    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"
Sep 16 19:11:35.276483 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.276449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt" event={"ID":"5c8da356-6f8f-4899-9787-a88f11c2bf1b","Type":"ContainerDied","Data":"8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14"}
Sep 16 19:11:35.276569 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.276516    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt" event={"ID":"5c8da356-6f8f-4899-9787-a88f11c2bf1b","Type":"ContainerDied","Data":"d051fc572041596d9e31386cca39c36877be4168d2418b1e03a1a69f1ce9c217"}
Sep 16 19:11:35.276569 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.276548    2559 scope.go:117] "RemoveContainer" containerID="8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14"
Sep 16 19:11:35.287116 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.287099    2559 scope.go:117] "RemoveContainer" containerID="8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14"
Sep 16 19:11:35.287437 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:11:35.287411    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14\": container with ID starting with 8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14 not found: ID does not exist" containerID="8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14"
Sep 16 19:11:35.287532 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.287445    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14"} err="failed to get container status \"8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14\": rpc error: code = NotFound desc = could not find container \"8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14\": container with ID starting with 8ed490d2bf3438cc5dd5fe74b11193653e963718694a1679bc932d4996d22a14 not found: ID does not exist"
Sep 16 19:11:35.324838 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.324805    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"]
Sep 16 19:11:35.351854 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.351821    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b04e50-7dbd5d75d5-6zsdt"]
Sep 16 19:11:35.353289 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.353269    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"]
Sep 16 19:11:35.357300 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.357276    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b04e50-p2zlv"]
Sep 16 19:11:35.369013 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.368992    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c8da356-6f8f-4899-9787-a88f11c2bf1b-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:35.369095 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.369016    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4f5v\" (UniqueName: \"kubernetes.io/projected/5c8da356-6f8f-4899-9787-a88f11c2bf1b-kube-api-access-r4f5v\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:35.381912 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.381890    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:35.725611 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.725553    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:35.930132 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:35.930082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:36.222434 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:36.222397    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55bd9a22-41de-4f99-817f-da2731c123e4" path="/var/lib/kubelet/pods/55bd9a22-41de-4f99-817f-da2731c123e4/volumes"
Sep 16 19:11:36.222847 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:36.222827    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8da356-6f8f-4899-9787-a88f11c2bf1b" path="/var/lib/kubelet/pods/5c8da356-6f8f-4899-9787-a88f11c2bf1b/volumes"
Sep 16 19:11:37.252683 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:37.252650    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:38.213113 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:38.213081    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:39.273295 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:39.273262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:40.232358 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:40.232328    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:41.270702 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:41.270671    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:43.362480 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.362441    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:43.467414 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.467380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:43.734733 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.734696    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:43.970793 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.970758    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"]
Sep 16 19:11:43.971128 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.971114    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55bd9a22-41de-4f99-817f-da2731c123e4" containerName="revoke-keys"
Sep 16 19:11:43.971128 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.971129    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="55bd9a22-41de-4f99-817f-da2731c123e4" containerName="revoke-keys"
Sep 16 19:11:43.971228 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.971148    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8da356-6f8f-4899-9787-a88f11c2bf1b" containerName="payload-processing"
Sep 16 19:11:43.971228 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.971154    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8da356-6f8f-4899-9787-a88f11c2bf1b" containerName="payload-processing"
Sep 16 19:11:43.971228 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.971204    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8da356-6f8f-4899-9787-a88f11c2bf1b" containerName="payload-processing"
Sep 16 19:11:43.971228 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.971212    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="55bd9a22-41de-4f99-817f-da2731c123e4" containerName="revoke-keys"
Sep 16 19:11:43.975374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.975357    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"
Sep 16 19:11:43.978127 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.978106    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:11:43.978243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.978107    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:11:43.986232 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:43.986173    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"]
Sep 16 19:11:44.043988 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:44.043939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940da8a2-8020-4ca3-a1ed-d4038cff5d70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm\" (UID: \"940da8a2-8020-4ca3-a1ed-d4038cff5d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"
Sep 16 19:11:44.145417 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:44.145380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940da8a2-8020-4ca3-a1ed-d4038cff5d70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm\" (UID: \"940da8a2-8020-4ca3-a1ed-d4038cff5d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"
Sep 16 19:11:44.145958 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:44.145939    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940da8a2-8020-4ca3-a1ed-d4038cff5d70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm\" (UID: \"940da8a2-8020-4ca3-a1ed-d4038cff5d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"
Sep 16 19:11:44.285121 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:44.285037    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"
Sep 16 19:11:44.311795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:44.311759    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:44.559358 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:44.559306    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"]
Sep 16 19:11:44.561500 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:11:44.561461    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940da8a2_8020_4ca3_a1ed_d4038cff5d70.slice/crio-edf89c21ca049c3886a56c76cc13605a6c7e5a343cdd8eb7dad187bcdd7d43ba WatchSource:0}: Error finding container edf89c21ca049c3886a56c76cc13605a6c7e5a343cdd8eb7dad187bcdd7d43ba: Status 404 returned error can't find the container with id edf89c21ca049c3886a56c76cc13605a6c7e5a343cdd8eb7dad187bcdd7d43ba
Sep 16 19:11:45.333585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:45.333553    2559 generic.go:356] "Generic (PLEG): container finished" podID="940da8a2-8020-4ca3-a1ed-d4038cff5d70" containerID="b0587f84cd884ce5b61ed36aa7fd5972ebc243f0a713dbed054059814aa01db8" exitCode=0
Sep 16 19:11:45.333743 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:45.333601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm" event={"ID":"940da8a2-8020-4ca3-a1ed-d4038cff5d70","Type":"ContainerDied","Data":"b0587f84cd884ce5b61ed36aa7fd5972ebc243f0a713dbed054059814aa01db8"}
Sep 16 19:11:45.333743 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:45.333638    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm" event={"ID":"940da8a2-8020-4ca3-a1ed-d4038cff5d70","Type":"ContainerStarted","Data":"edf89c21ca049c3886a56c76cc13605a6c7e5a343cdd8eb7dad187bcdd7d43ba"}
Sep 16 19:11:45.357603 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:45.357574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:46.351258 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:46.351233    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:46.468540 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:46.468517    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"
Sep 16 19:11:46.564322 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:46.564281    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940da8a2-8020-4ca3-a1ed-d4038cff5d70-maas-api-service-ca\") pod \"940da8a2-8020-4ca3-a1ed-d4038cff5d70\" (UID: \"940da8a2-8020-4ca3-a1ed-d4038cff5d70\") "
Sep 16 19:11:46.564642 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:46.564620    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940da8a2-8020-4ca3-a1ed-d4038cff5d70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "940da8a2-8020-4ca3-a1ed-d4038cff5d70" (UID: "940da8a2-8020-4ca3-a1ed-d4038cff5d70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:46.665786 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:46.665693    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940da8a2-8020-4ca3-a1ed-d4038cff5d70-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:47.199377 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:47.199334    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:47.343507 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:47.343479    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"
Sep 16 19:11:47.343653 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:47.343476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm" event={"ID":"940da8a2-8020-4ca3-a1ed-d4038cff5d70","Type":"ContainerDied","Data":"edf89c21ca049c3886a56c76cc13605a6c7e5a343cdd8eb7dad187bcdd7d43ba"}
Sep 16 19:11:47.343653 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:47.343611    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf89c21ca049c3886a56c76cc13605a6c7e5a343cdd8eb7dad187bcdd7d43ba"
Sep 16 19:11:48.069863 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:48.069828    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:49.121112 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:49.121076    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:49.943843 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:49.943815    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:54.736780 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:54.736719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:54.800992 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:54.800947    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"]
Sep 16 19:11:54.801228 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:54.801200    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n" podUID="d973095d-db08-41a4-af4d-74b4410606b9" containerName="payload-processing" containerID="cri-o://8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc" gracePeriod=30
Sep 16 19:11:54.829280 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:54.829230    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:54.897500 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:54.897463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:54.990772 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:54.990705    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"]
Sep 16 19:11:54.994758 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:54.994724    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c20219-qjklm"]
Sep 16 19:11:54.994891 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:54.994830    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:55.050472 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.050452    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:11:55.079918 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.079894    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:55.143241 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.143207    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h42t\" (UniqueName: \"kubernetes.io/projected/d973095d-db08-41a4-af4d-74b4410606b9-kube-api-access-9h42t\") pod \"d973095d-db08-41a4-af4d-74b4410606b9\" (UID: \"d973095d-db08-41a4-af4d-74b4410606b9\") "
Sep 16 19:11:55.143401 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.143291    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973095d-db08-41a4-af4d-74b4410606b9-plugins-config-volume\") pod \"d973095d-db08-41a4-af4d-74b4410606b9\" (UID: \"d973095d-db08-41a4-af4d-74b4410606b9\") "
Sep 16 19:11:55.143693 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.143670    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d973095d-db08-41a4-af4d-74b4410606b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d973095d-db08-41a4-af4d-74b4410606b9" (UID: "d973095d-db08-41a4-af4d-74b4410606b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:11:55.145392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.145370    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d973095d-db08-41a4-af4d-74b4410606b9-kube-api-access-9h42t" (OuterVolumeSpecName: "kube-api-access-9h42t") pod "d973095d-db08-41a4-af4d-74b4410606b9" (UID: "d973095d-db08-41a4-af4d-74b4410606b9"). InnerVolumeSpecName "kube-api-access-9h42t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:55.244411 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.244332    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h42t\" (UniqueName: \"kubernetes.io/projected/d973095d-db08-41a4-af4d-74b4410606b9-kube-api-access-9h42t\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.244411 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.244360    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973095d-db08-41a4-af4d-74b4410606b9-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:11:55.351407 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.351376    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:55.382613 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.382581    2559 generic.go:356] "Generic (PLEG): container finished" podID="d973095d-db08-41a4-af4d-74b4410606b9" containerID="8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc" exitCode=0
Sep 16 19:11:55.382781 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.382639    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"
Sep 16 19:11:55.382781 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.382678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n" event={"ID":"d973095d-db08-41a4-af4d-74b4410606b9","Type":"ContainerDied","Data":"8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc"}
Sep 16 19:11:55.382781 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.382731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n" event={"ID":"d973095d-db08-41a4-af4d-74b4410606b9","Type":"ContainerDied","Data":"c54ceb4f40cd65857e901c11eeb347f0d075fef39edd4718786cb96e1295d66a"}
Sep 16 19:11:55.382781 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.382752    2559 scope.go:117] "RemoveContainer" containerID="8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc"
Sep 16 19:11:55.391367 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.391349    2559 scope.go:117] "RemoveContainer" containerID="8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc"
Sep 16 19:11:55.391657 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:11:55.391637    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc\": container with ID starting with 8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc not found: ID does not exist" containerID="8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc"
Sep 16 19:11:55.391738 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.391669    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc"} err="failed to get container status \"8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc\": rpc error: code = NotFound desc = could not find container \"8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc\": container with ID starting with 8696d8a738e4e048b878a66fcb51857a66bef512ac37ebe7fa276edabf0cf9fc not found: ID does not exist"
Sep 16 19:11:55.408447 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.408421    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"]
Sep 16 19:11:55.414066 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.414041    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c20219-587fbd68d7-rm94n"]
Sep 16 19:11:55.453652 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:55.453624    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:56.222231 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:56.222198    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="940da8a2-8020-4ca3-a1ed-d4038cff5d70" path="/var/lib/kubelet/pods/940da8a2-8020-4ca3-a1ed-d4038cff5d70/volumes"
Sep 16 19:11:56.222600 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:56.222545    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d973095d-db08-41a4-af4d-74b4410606b9" path="/var/lib/kubelet/pods/d973095d-db08-41a4-af4d-74b4410606b9/volumes"
Sep 16 19:11:56.310958 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:56.310927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:57.505160 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:57.505124    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:57.939172 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:57.939059    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:58.038301 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:58.038269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:59.006718 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:59.006684    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:59.348875 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:11:59.348788    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:00.380606 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:00.380575    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:02.743556 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:02.743515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:03.320458 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.320418    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"]
Sep 16 19:12:03.320828 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.320810    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940da8a2-8020-4ca3-a1ed-d4038cff5d70" containerName="revoke-keys"
Sep 16 19:12:03.320923 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.320830    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="940da8a2-8020-4ca3-a1ed-d4038cff5d70" containerName="revoke-keys"
Sep 16 19:12:03.320923 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.320844    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d973095d-db08-41a4-af4d-74b4410606b9" containerName="payload-processing"
Sep 16 19:12:03.320923 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.320853    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d973095d-db08-41a4-af4d-74b4410606b9" containerName="payload-processing"
Sep 16 19:12:03.321101 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.320960    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="940da8a2-8020-4ca3-a1ed-d4038cff5d70" containerName="revoke-keys"
Sep 16 19:12:03.321101 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.320989    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d973095d-db08-41a4-af4d-74b4410606b9" containerName="payload-processing"
Sep 16 19:12:03.323831 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.323809    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"
Sep 16 19:12:03.326222 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.326199    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:03.327493 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.327475    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:12:03.327665 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.327646    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:12:03.335750 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.335724    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"]
Sep 16 19:12:03.400735 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.400706    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:03.417864 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.417837    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba71df-e8ab-473b-9749-b408697b5b12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8\" (UID: \"43ba71df-e8ab-473b-9749-b408697b5b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"
Sep 16 19:12:03.518622 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.518578    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba71df-e8ab-473b-9749-b408697b5b12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8\" (UID: \"43ba71df-e8ab-473b-9749-b408697b5b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"
Sep 16 19:12:03.519241 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.519222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba71df-e8ab-473b-9749-b408697b5b12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8\" (UID: \"43ba71df-e8ab-473b-9749-b408697b5b12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"
Sep 16 19:12:03.633346 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.633263    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"
Sep 16 19:12:03.639320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.639291    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:03.878059 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.878028    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:03.903191 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:03.903166    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"]
Sep 16 19:12:03.905277 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:12:03.905252    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ba71df_e8ab_473b_9749_b408697b5b12.slice/crio-94136c517229264a8652983a496a6eb9ea5f3a506499991a6ae51372f5ba8987 WatchSource:0}: Error finding container 94136c517229264a8652983a496a6eb9ea5f3a506499991a6ae51372f5ba8987: Status 404 returned error can't find the container with id 94136c517229264a8652983a496a6eb9ea5f3a506499991a6ae51372f5ba8987
Sep 16 19:12:04.424911 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:04.424810    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8" event={"ID":"43ba71df-e8ab-473b-9749-b408697b5b12","Type":"ContainerStarted","Data":"ecd2a79e3194f10d22000a8b6ff6aef712a6311c183595f14365d55554a73c51"}
Sep 16 19:12:04.424911 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:04.424862    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8" event={"ID":"43ba71df-e8ab-473b-9749-b408697b5b12","Type":"ContainerStarted","Data":"94136c517229264a8652983a496a6eb9ea5f3a506499991a6ae51372f5ba8987"}
Sep 16 19:12:04.446581 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:04.446533    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8" podStartSLOduration=1.446517025 podStartE2EDuration="1.446517025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:04.443989275 +0000 UTC m=+1956.812045193" watchObservedRunningTime="2026-09-16 19:12:04.446517025 +0000 UTC m=+1956.814572925"
Sep 16 19:12:04.629402 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:04.629369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:05.431392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:05.431358    2559 generic.go:356] "Generic (PLEG): container finished" podID="43ba71df-e8ab-473b-9749-b408697b5b12" containerID="ecd2a79e3194f10d22000a8b6ff6aef712a6311c183595f14365d55554a73c51" exitCode=0
Sep 16 19:12:05.431782 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:05.431410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8" event={"ID":"43ba71df-e8ab-473b-9749-b408697b5b12","Type":"ContainerDied","Data":"ecd2a79e3194f10d22000a8b6ff6aef712a6311c183595f14365d55554a73c51"}
Sep 16 19:12:05.518020 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:05.517986    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:06.345251 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:06.345216    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:06.565207 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:06.565180    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"
Sep 16 19:12:06.646200 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:06.646113    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba71df-e8ab-473b-9749-b408697b5b12-maas-api-service-ca\") pod \"43ba71df-e8ab-473b-9749-b408697b5b12\" (UID: \"43ba71df-e8ab-473b-9749-b408697b5b12\") "
Sep 16 19:12:06.646471 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:06.646448    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ba71df-e8ab-473b-9749-b408697b5b12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43ba71df-e8ab-473b-9749-b408697b5b12" (UID: "43ba71df-e8ab-473b-9749-b408697b5b12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:06.747438 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:06.747400    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ba71df-e8ab-473b-9749-b408697b5b12-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:07.345511 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:07.345478    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:07.442601 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:07.442575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"
Sep 16 19:12:07.442772 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:07.442609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8" event={"ID":"43ba71df-e8ab-473b-9749-b408697b5b12","Type":"ContainerDied","Data":"94136c517229264a8652983a496a6eb9ea5f3a506499991a6ae51372f5ba8987"}
Sep 16 19:12:07.442772 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:07.442658    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94136c517229264a8652983a496a6eb9ea5f3a506499991a6ae51372f5ba8987"
Sep 16 19:12:08.056434 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:08.056389    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:14.136330 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.136301    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:14.167017 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.166989    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"]
Sep 16 19:12:14.167220 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.167196    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6" podUID="1a30f659-1519-496e-b079-fd318ab77dad" containerName="payload-processing" containerID="cri-o://b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714" gracePeriod=30
Sep 16 19:12:14.242842 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.242798    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:14.343263 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.343237    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:14.417589 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.417565    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:12:14.462426 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.462388    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"]
Sep 16 19:12:14.466455 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.466416    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:14.466680 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.466661    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58bf51-cclj8"]
Sep 16 19:12:14.474391 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.474366    2559 generic.go:356] "Generic (PLEG): container finished" podID="1a30f659-1519-496e-b079-fd318ab77dad" containerID="b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714" exitCode=0
Sep 16 19:12:14.474482 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.474411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6" event={"ID":"1a30f659-1519-496e-b079-fd318ab77dad","Type":"ContainerDied","Data":"b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714"}
Sep 16 19:12:14.474482 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.474428    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"
Sep 16 19:12:14.474555 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.474487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6" event={"ID":"1a30f659-1519-496e-b079-fd318ab77dad","Type":"ContainerDied","Data":"204891da5c4a13ee4d5d816299389ca420c911b4ccebd2f426322ddae49e7e8f"}
Sep 16 19:12:14.474555 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.474510    2559 scope.go:117] "RemoveContainer" containerID="b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714"
Sep 16 19:12:14.482823 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.482797    2559 scope.go:117] "RemoveContainer" containerID="b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714"
Sep 16 19:12:14.483158 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:14.483132    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714\": container with ID starting with b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714 not found: ID does not exist" containerID="b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714"
Sep 16 19:12:14.483243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.483167    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714"} err="failed to get container status \"b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714\": rpc error: code = NotFound desc = could not find container \"b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714\": container with ID starting with b2dbfb2087999a7c3676cda2aec445c02afcf850426a8e5f64cb9305d40a1714 not found: ID does not exist"
Sep 16 19:12:14.518257 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.518231    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a30f659-1519-496e-b079-fd318ab77dad-plugins-config-volume\") pod \"1a30f659-1519-496e-b079-fd318ab77dad\" (UID: \"1a30f659-1519-496e-b079-fd318ab77dad\") "
Sep 16 19:12:14.518388 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.518371    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtxlc\" (UniqueName: \"kubernetes.io/projected/1a30f659-1519-496e-b079-fd318ab77dad-kube-api-access-rtxlc\") pod \"1a30f659-1519-496e-b079-fd318ab77dad\" (UID: \"1a30f659-1519-496e-b079-fd318ab77dad\") "
Sep 16 19:12:14.518599 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.518574    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a30f659-1519-496e-b079-fd318ab77dad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a30f659-1519-496e-b079-fd318ab77dad" (UID: "1a30f659-1519-496e-b079-fd318ab77dad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:14.520431 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.520408    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a30f659-1519-496e-b079-fd318ab77dad-kube-api-access-rtxlc" (OuterVolumeSpecName: "kube-api-access-rtxlc") pod "1a30f659-1519-496e-b079-fd318ab77dad" (UID: "1a30f659-1519-496e-b079-fd318ab77dad"). InnerVolumeSpecName "kube-api-access-rtxlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:14.619339 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.619307    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtxlc\" (UniqueName: \"kubernetes.io/projected/1a30f659-1519-496e-b079-fd318ab77dad-kube-api-access-rtxlc\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.619339 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.619338    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a30f659-1519-496e-b079-fd318ab77dad-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:14.800776 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.800743    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"]
Sep 16 19:12:14.821846 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.821810    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58bf51-6b6bc69869-86jz6"]
Sep 16 19:12:14.825857 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.825830    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:14.940744 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:14.940714    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:15.914720 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:15.914686    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:16.223225 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:16.223191    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a30f659-1519-496e-b079-fd318ab77dad" path="/var/lib/kubelet/pods/1a30f659-1519-496e-b079-fd318ab77dad/volumes"
Sep 16 19:12:16.223591 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:16.223575    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ba71df-e8ab-473b-9749-b408697b5b12" path="/var/lib/kubelet/pods/43ba71df-e8ab-473b-9749-b408697b5b12/volumes"
Sep 16 19:12:16.630497 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:16.630390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:17.648594 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:17.648561    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:21.324281 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:21.324247    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:21.388651 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:21.388624    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:21.617158 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:21.617082    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:28.783310 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.783267    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:28.889919 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.889880    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"]
Sep 16 19:12:28.890229 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.890216    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a30f659-1519-496e-b079-fd318ab77dad" containerName="payload-processing"
Sep 16 19:12:28.890279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.890230    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a30f659-1519-496e-b079-fd318ab77dad" containerName="payload-processing"
Sep 16 19:12:28.890279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.890247    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ba71df-e8ab-473b-9749-b408697b5b12" containerName="revoke-keys"
Sep 16 19:12:28.890279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.890253    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ba71df-e8ab-473b-9749-b408697b5b12" containerName="revoke-keys"
Sep 16 19:12:28.890384 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.890313    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ba71df-e8ab-473b-9749-b408697b5b12" containerName="revoke-keys"
Sep 16 19:12:28.890384 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.890324    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a30f659-1519-496e-b079-fd318ab77dad" containerName="payload-processing"
Sep 16 19:12:28.894766 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.894747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"
Sep 16 19:12:28.897986 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.897957    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:12:28.898312 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.898286    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:12:28.903239 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:28.903217    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"]
Sep 16 19:12:29.039301 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:29.039218    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020c0646-da68-4c66-bada-8052a10ae2c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw\" (UID: \"020c0646-da68-4c66-bada-8052a10ae2c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"
Sep 16 19:12:29.140601 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:29.140567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020c0646-da68-4c66-bada-8052a10ae2c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw\" (UID: \"020c0646-da68-4c66-bada-8052a10ae2c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"
Sep 16 19:12:29.141196 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:29.141174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020c0646-da68-4c66-bada-8052a10ae2c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw\" (UID: \"020c0646-da68-4c66-bada-8052a10ae2c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"
Sep 16 19:12:29.206230 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:29.206199    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"
Sep 16 19:12:29.211431 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:29.211409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:29.479320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:29.479285    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"]
Sep 16 19:12:29.482219 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:12:29.482183    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020c0646_da68_4c66_bada_8052a10ae2c3.slice/crio-04fc1656c1ac51ea233bbfdba2162f92041f3d2d86e54b23c5cd16abdb6087ab WatchSource:0}: Error finding container 04fc1656c1ac51ea233bbfdba2162f92041f3d2d86e54b23c5cd16abdb6087ab: Status 404 returned error can't find the container with id 04fc1656c1ac51ea233bbfdba2162f92041f3d2d86e54b23c5cd16abdb6087ab
Sep 16 19:12:29.548404 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:29.548365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw" event={"ID":"020c0646-da68-4c66-bada-8052a10ae2c3","Type":"ContainerStarted","Data":"04fc1656c1ac51ea233bbfdba2162f92041f3d2d86e54b23c5cd16abdb6087ab"}
Sep 16 19:12:29.973132 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:29.973094    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:30.553937 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:30.553900    2559 generic.go:356] "Generic (PLEG): container finished" podID="020c0646-da68-4c66-bada-8052a10ae2c3" containerID="98711ff655a30b2962d88e71764796ee7c703a211b5fe59697c870c7cfb867ad" exitCode=0
Sep 16 19:12:30.554121 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:30.553997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw" event={"ID":"020c0646-da68-4c66-bada-8052a10ae2c3","Type":"ContainerDied","Data":"98711ff655a30b2962d88e71764796ee7c703a211b5fe59697c870c7cfb867ad"}
Sep 16 19:12:30.828455 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:30.828369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:31.623783 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:31.623757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:31.687911 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:31.687885    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"
Sep 16 19:12:31.761931 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:31.761905    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020c0646-da68-4c66-bada-8052a10ae2c3-maas-api-service-ca\") pod \"020c0646-da68-4c66-bada-8052a10ae2c3\" (UID: \"020c0646-da68-4c66-bada-8052a10ae2c3\") "
Sep 16 19:12:31.762279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:31.762255    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020c0646-da68-4c66-bada-8052a10ae2c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "020c0646-da68-4c66-bada-8052a10ae2c3" (UID: "020c0646-da68-4c66-bada-8052a10ae2c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:31.863198 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:31.863116    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020c0646-da68-4c66-bada-8052a10ae2c3-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:32.565254 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:32.565164    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw" event={"ID":"020c0646-da68-4c66-bada-8052a10ae2c3","Type":"ContainerDied","Data":"04fc1656c1ac51ea233bbfdba2162f92041f3d2d86e54b23c5cd16abdb6087ab"}
Sep 16 19:12:32.565254 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:32.565191    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"
Sep 16 19:12:32.565254 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:32.565212    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04fc1656c1ac51ea233bbfdba2162f92041f3d2d86e54b23c5cd16abdb6087ab"
Sep 16 19:12:32.851924 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:32.851833    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:33.764225 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:33.764191    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"]
Sep 16 19:12:33.800587 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:33.800547    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"]
Sep 16 19:12:33.915418 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:33.915386    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:45.116614 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.116577    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"]
Sep 16 19:12:45.118147 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.118122    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c090c8-29826420-mvwdk"]
Sep 16 19:12:45.148002 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.147952    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"]
Sep 16 19:12:45.148308 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.148259    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs" podUID="6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" containerName="payload-processing" containerID="cri-o://98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2" gracePeriod=30
Sep 16 19:12:45.247908 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.247872    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:45.283494 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.283448    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs" podUID="6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.34:9004: connect: connection refused"
Sep 16 19:12:45.347261 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.347232    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:45.366953 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.366880    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"]
Sep 16 19:12:45.372414 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.372387    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c090c8-s26kw"]
Sep 16 19:12:45.405616 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.405594    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 19:12:45.428856 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.428825    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:45.483240 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.483213    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-plugins-config-volume\") pod \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\" (UID: \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\") "
Sep 16 19:12:45.483410 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.483252    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn4b8\" (UniqueName: \"kubernetes.io/projected/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-kube-api-access-tn4b8\") pod \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\" (UID: \"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e\") "
Sep 16 19:12:45.483537 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.483509    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" (UID: "6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:45.485305 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.485283    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-kube-api-access-tn4b8" (OuterVolumeSpecName: "kube-api-access-tn4b8") pod "6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" (UID: "6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e"). InnerVolumeSpecName "kube-api-access-tn4b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:45.584524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.584484    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.584524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.584517    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4b8\" (UniqueName: \"kubernetes.io/projected/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e-kube-api-access-tn4b8\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:45.625921 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.625828    2559 generic.go:356] "Generic (PLEG): container finished" podID="6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" containerID="98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2" exitCode=0
Sep 16 19:12:45.625921 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.625879    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"
Sep 16 19:12:45.626137 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.625917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs" event={"ID":"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e","Type":"ContainerDied","Data":"98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2"}
Sep 16 19:12:45.626137 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.625992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs" event={"ID":"6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e","Type":"ContainerDied","Data":"b52766b5d36071046ad0d45480c48049349d87ffcb20750b051ef2b4e8a3c2cf"}
Sep 16 19:12:45.626137 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.626017    2559 scope.go:117] "RemoveContainer" containerID="98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2"
Sep 16 19:12:45.634726 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.634712    2559 scope.go:117] "RemoveContainer" containerID="98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2"
Sep 16 19:12:45.635004 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:45.634984    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2\": container with ID starting with 98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2 not found: ID does not exist" containerID="98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2"
Sep 16 19:12:45.635071 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.635010    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2"} err="failed to get container status \"98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2\": rpc error: code = NotFound desc = could not find container \"98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2\": container with ID starting with 98099f911f3cc2f4ba6a2bfbf7c5aa745d4e1a763974210676f6d06e671673a2 not found: ID does not exist"
Sep 16 19:12:45.647622 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.647601    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"]
Sep 16 19:12:45.651594 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.651571    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c090c8-6dd8f65fd9-tq6hs"]
Sep 16 19:12:45.754956 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.754927    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:45.818309 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:45.818272    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:46.222304 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.222268    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020c0646-da68-4c66-bada-8052a10ae2c3" path="/var/lib/kubelet/pods/020c0646-da68-4c66-bada-8052a10ae2c3/volumes"
Sep 16 19:12:46.222672 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.222542    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dcfbf7f-4c20-44be-98d2-f26a184b28fb" path="/var/lib/kubelet/pods/5dcfbf7f-4c20-44be-98d2-f26a184b28fb/volumes"
Sep 16 19:12:46.222816 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.222803    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" path="/var/lib/kubelet/pods/6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e/volumes"
Sep 16 19:12:46.622917 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.622829    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:46.965293 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.965242    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7226-433364"]
Sep 16 19:12:46.965631 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.965613    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020c0646-da68-4c66-bada-8052a10ae2c3" containerName="revoke-keys"
Sep 16 19:12:46.965715 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.965634    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="020c0646-da68-4c66-bada-8052a10ae2c3" containerName="revoke-keys"
Sep 16 19:12:46.965715 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.965671    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" containerName="payload-processing"
Sep 16 19:12:46.965715 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.965680    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" containerName="payload-processing"
Sep 16 19:12:46.965876 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.965759    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eda8e0f-ae0f-49d7-aa61-3d3c82d43f9e" containerName="payload-processing"
Sep 16 19:12:46.965876 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.965773    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="020c0646-da68-4c66-bada-8052a10ae2c3" containerName="revoke-keys"
Sep 16 19:12:46.970443 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.970419    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:46.973375 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.973355    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q2jrz\""
Sep 16 19:12:46.976565 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:46.976505    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7226-433364"]
Sep 16 19:12:47.098830 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.098793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdc4\" (UniqueName: \"kubernetes.io/projected/a194673e-27c9-4ae9-bc0b-868a812a0b12-kube-api-access-scdc4\") pod \"test-curl-7226-433364\" (UID: \"a194673e-27c9-4ae9-bc0b-868a812a0b12\") " pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:47.200031 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.199957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdc4\" (UniqueName: \"kubernetes.io/projected/a194673e-27c9-4ae9-bc0b-868a812a0b12-kube-api-access-scdc4\") pod \"test-curl-7226-433364\" (UID: \"a194673e-27c9-4ae9-bc0b-868a812a0b12\") " pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:47.215209 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.215174    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdc4\" (UniqueName: \"kubernetes.io/projected/a194673e-27c9-4ae9-bc0b-868a812a0b12-kube-api-access-scdc4\") pod \"test-curl-7226-433364\" (UID: \"a194673e-27c9-4ae9-bc0b-868a812a0b12\") " pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:47.281091 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.280969    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:47.560502 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.560305    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7226-433364"]
Sep 16 19:12:47.563133 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:12:47.563107    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda194673e_27c9_4ae9_bc0b_868a812a0b12.slice/crio-483cb9daf163c5d70a8fa519fe075bb5ccf0fc76c5f66094db140c5bb810494b WatchSource:0}: Error finding container 483cb9daf163c5d70a8fa519fe075bb5ccf0fc76c5f66094db140c5bb810494b: Status 404 returned error can't find the container with id 483cb9daf163c5d70a8fa519fe075bb5ccf0fc76c5f66094db140c5bb810494b
Sep 16 19:12:47.638209 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.638171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7226-433364" event={"ID":"a194673e-27c9-4ae9-bc0b-868a812a0b12","Type":"ContainerStarted","Data":"77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704"}
Sep 16 19:12:47.638346 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.638212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7226-433364" event={"ID":"a194673e-27c9-4ae9-bc0b-868a812a0b12","Type":"ContainerStarted","Data":"483cb9daf163c5d70a8fa519fe075bb5ccf0fc76c5f66094db140c5bb810494b"}
Sep 16 19:12:47.656627 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.656581    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7226-433364" podStartSLOduration=1.6565665840000001 podStartE2EDuration="1.656566584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:47.655432314 +0000 UTC m=+2000.023488215" watchObservedRunningTime="2026-09-16 19:12:47.656566584 +0000 UTC m=+2000.024622537"
Sep 16 19:12:47.943243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:47.943214    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:48.418964 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:48.418869    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7226-433364"]
Sep 16 19:12:48.449812 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:48.445847    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7226-433364"]
Sep 16 19:12:48.642202 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:48.642173    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7226-433364" container="test-curl-7226-433364"
Sep 16 19:12:48.783097 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:48.783062    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6" podUID="376ea665-248b-42c0-b4e4-9c4864d5d3b1" containerName="main" containerID="cri-o://4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3" gracePeriod=45
Sep 16 19:12:48.818611 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:48.818574    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2" podUID="7099c17b-763b-4c64-b8af-3c64de776e64" containerName="main" containerID="cri-o://3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d" gracePeriod=45
Sep 16 19:12:49.052235 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.052211    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:12:49.074049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.074021    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 19:12:49.218503 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218472    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzkzb\" (UniqueName: \"kubernetes.io/projected/376ea665-248b-42c0-b4e4-9c4864d5d3b1-kube-api-access-rzkzb\") pod \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") "
Sep 16 19:12:49.218670 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218540    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-model-cache\") pod \"7099c17b-763b-4c64-b8af-3c64de776e64\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") "
Sep 16 19:12:49.218670 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-home\") pod \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") "
Sep 16 19:12:49.218805 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218682    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-tmp-dir\") pod \"7099c17b-763b-4c64-b8af-3c64de776e64\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") "
Sep 16 19:12:49.218805 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218700    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-dshm\") pod \"7099c17b-763b-4c64-b8af-3c64de776e64\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") "
Sep 16 19:12:49.218805 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218742    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkv4d\" (UniqueName: \"kubernetes.io/projected/7099c17b-763b-4c64-b8af-3c64de776e64-kube-api-access-wkv4d\") pod \"7099c17b-763b-4c64-b8af-3c64de776e64\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") "
Sep 16 19:12:49.218805 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218800    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7099c17b-763b-4c64-b8af-3c64de776e64-tls-certs\") pod \"7099c17b-763b-4c64-b8af-3c64de776e64\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") "
Sep 16 19:12:49.219049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218802    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-model-cache" (OuterVolumeSpecName: "model-cache") pod "7099c17b-763b-4c64-b8af-3c64de776e64" (UID: "7099c17b-763b-4c64-b8af-3c64de776e64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.219049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218826    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-model-cache\") pod \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") "
Sep 16 19:12:49.219049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218851    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-dshm\") pod \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") "
Sep 16 19:12:49.219049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218888    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-home\") pod \"7099c17b-763b-4c64-b8af-3c64de776e64\" (UID: \"7099c17b-763b-4c64-b8af-3c64de776e64\") "
Sep 16 19:12:49.219049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218911    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tls-certs\") pod \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") "
Sep 16 19:12:49.219049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218933    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-home" (OuterVolumeSpecName: "home") pod "376ea665-248b-42c0-b4e4-9c4864d5d3b1" (UID: "376ea665-248b-42c0-b4e4-9c4864d5d3b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.219049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218940    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7099c17b-763b-4c64-b8af-3c64de776e64" (UID: "7099c17b-763b-4c64-b8af-3c64de776e64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.219049 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.218944    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tmp-dir\") pod \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\" (UID: \"376ea665-248b-42c0-b4e4-9c4864d5d3b1\") "
Sep 16 19:12:49.219464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.219177    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "376ea665-248b-42c0-b4e4-9c4864d5d3b1" (UID: "376ea665-248b-42c0-b4e4-9c4864d5d3b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.219464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.219178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-home" (OuterVolumeSpecName: "home") pod "7099c17b-763b-4c64-b8af-3c64de776e64" (UID: "7099c17b-763b-4c64-b8af-3c64de776e64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.219464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.219192    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "376ea665-248b-42c0-b4e4-9c4864d5d3b1" (UID: "376ea665-248b-42c0-b4e4-9c4864d5d3b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.219464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.219245    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.219464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.219319    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.219464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.219361    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.219464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.219377    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.219464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.219391    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.221516 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.221492    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-dshm" (OuterVolumeSpecName: "dshm") pod "7099c17b-763b-4c64-b8af-3c64de776e64" (UID: "7099c17b-763b-4c64-b8af-3c64de776e64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.221680 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.221662    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7099c17b-763b-4c64-b8af-3c64de776e64-kube-api-access-wkv4d" (OuterVolumeSpecName: "kube-api-access-wkv4d") pod "7099c17b-763b-4c64-b8af-3c64de776e64" (UID: "7099c17b-763b-4c64-b8af-3c64de776e64"). InnerVolumeSpecName "kube-api-access-wkv4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:49.221680 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.221667    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-dshm" (OuterVolumeSpecName: "dshm") pod "376ea665-248b-42c0-b4e4-9c4864d5d3b1" (UID: "376ea665-248b-42c0-b4e4-9c4864d5d3b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:49.221795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.221662    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7099c17b-763b-4c64-b8af-3c64de776e64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7099c17b-763b-4c64-b8af-3c64de776e64" (UID: "7099c17b-763b-4c64-b8af-3c64de776e64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:49.221795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.221690    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376ea665-248b-42c0-b4e4-9c4864d5d3b1-kube-api-access-rzkzb" (OuterVolumeSpecName: "kube-api-access-rzkzb") pod "376ea665-248b-42c0-b4e4-9c4864d5d3b1" (UID: "376ea665-248b-42c0-b4e4-9c4864d5d3b1"). InnerVolumeSpecName "kube-api-access-rzkzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:49.221891 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.221875    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "376ea665-248b-42c0-b4e4-9c4864d5d3b1" (UID: "376ea665-248b-42c0-b4e4-9c4864d5d3b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:49.320205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.320109    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7099c17b-763b-4c64-b8af-3c64de776e64-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.320205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.320140    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkv4d\" (UniqueName: \"kubernetes.io/projected/7099c17b-763b-4c64-b8af-3c64de776e64-kube-api-access-wkv4d\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.320205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.320154    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7099c17b-763b-4c64-b8af-3c64de776e64-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.320205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.320166    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.320205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.320176    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/376ea665-248b-42c0-b4e4-9c4864d5d3b1-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.320205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.320186    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/376ea665-248b-42c0-b4e4-9c4864d5d3b1-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.320205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.320197    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzkzb\" (UniqueName: \"kubernetes.io/projected/376ea665-248b-42c0-b4e4-9c4864d5d3b1-kube-api-access-rzkzb\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:49.648027 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.647919    2559 generic.go:356] "Generic (PLEG): container finished" podID="7099c17b-763b-4c64-b8af-3c64de776e64" containerID="3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d" exitCode=0
Sep 16 19:12:49.648027 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.647960    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"
Sep 16 19:12:49.648027 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.648007    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2" event={"ID":"7099c17b-763b-4c64-b8af-3c64de776e64","Type":"ContainerDied","Data":"3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d"}
Sep 16 19:12:49.648557 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.648055    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2" event={"ID":"7099c17b-763b-4c64-b8af-3c64de776e64","Type":"ContainerDied","Data":"1466d95e0965488a6d754e2aa0590e690ccc8f1131202edbeee33b736b55e0cc"}
Sep 16 19:12:49.648557 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.648073    2559 scope.go:117] "RemoveContainer" containerID="3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d"
Sep 16 19:12:49.649519 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.649487    2559 generic.go:356] "Generic (PLEG): container finished" podID="376ea665-248b-42c0-b4e4-9c4864d5d3b1" containerID="4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3" exitCode=0
Sep 16 19:12:49.649646 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.649540    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"
Sep 16 19:12:49.649646 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.649577    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6" event={"ID":"376ea665-248b-42c0-b4e4-9c4864d5d3b1","Type":"ContainerDied","Data":"4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3"}
Sep 16 19:12:49.649646 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.649607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6" event={"ID":"376ea665-248b-42c0-b4e4-9c4864d5d3b1","Type":"ContainerDied","Data":"4b5f7860a5c44c0bfdbb491dcf69f1be206f3e949a7c8479b0e4cdecc0220671"}
Sep 16 19:12:49.649805 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.649787    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7226-433364" podUID="a194673e-27c9-4ae9-bc0b-868a812a0b12" containerName="test-curl-7226-433364" containerID="cri-o://77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704" gracePeriod=2
Sep 16 19:12:49.657966 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:49.657939    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7226-433364\" is forbidden: User \"system:node:ip-10-0-141-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-134.ec2.internal' and this object" podUID="a194673e-27c9-4ae9-bc0b-868a812a0b12" pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:49.658630 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.658612    2559 scope.go:117] "RemoveContainer" containerID="3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d"
Sep 16 19:12:49.658908 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:49.658889    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d\": container with ID starting with 3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d not found: ID does not exist" containerID="3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d"
Sep 16 19:12:49.659024 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.658920    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d"} err="failed to get container status \"3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d\": rpc error: code = NotFound desc = could not find container \"3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d\": container with ID starting with 3bfcafb43dfae38d6928e6136ca2c19d03964a7087c8ff9a8bbc6ec94738250d not found: ID does not exist"
Sep 16 19:12:49.659024 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.658941    2559 scope.go:117] "RemoveContainer" containerID="4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3"
Sep 16 19:12:49.666776 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.666752    2559 scope.go:117] "RemoveContainer" containerID="4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3"
Sep 16 19:12:49.667058 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:49.667039    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3\": container with ID starting with 4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3 not found: ID does not exist" containerID="4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3"
Sep 16 19:12:49.667117 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.667062    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3"} err="failed to get container status \"4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3\": rpc error: code = NotFound desc = could not find container \"4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3\": container with ID starting with 4184533a9b52c01c0971b0fdccd07de99d3406bc208225904c657ff4708d41f3 not found: ID does not exist"
Sep 16 19:12:49.677643 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.677623    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"]
Sep 16 19:12:49.680592 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.680572    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/distinct-2-w4-c090c8-kserve-7d55c48d49-678m6"]
Sep 16 19:12:49.694657 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.694639    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"]
Sep 16 19:12:49.711883 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:49.711864    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c090c8/simulator-w4-c090c8-kserve-64c4dfc8bb-2zmj2"]
Sep 16 19:12:49.713152 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:49.713133    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7226-433364\" is forbidden: User \"system:node:ip-10-0-141-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-134.ec2.internal' and this object" podUID="a194673e-27c9-4ae9-bc0b-868a812a0b12" pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:49.715503 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:49.715484    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7226-433364\" is forbidden: User \"system:node:ip-10-0-141-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-134.ec2.internal' and this object" podUID="a194673e-27c9-4ae9-bc0b-868a812a0b12" pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:50.222272 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:50.222238    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="376ea665-248b-42c0-b4e4-9c4864d5d3b1" path="/var/lib/kubelet/pods/376ea665-248b-42c0-b4e4-9c4864d5d3b1/volumes"
Sep 16 19:12:50.222644 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:50.222630    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7099c17b-763b-4c64-b8af-3c64de776e64" path="/var/lib/kubelet/pods/7099c17b-763b-4c64-b8af-3c64de776e64/volumes"
Sep 16 19:12:50.722895 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:50.722858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:51.835953 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:51.835930    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:51.942442 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:51.942361    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scdc4\" (UniqueName: \"kubernetes.io/projected/a194673e-27c9-4ae9-bc0b-868a812a0b12-kube-api-access-scdc4\") pod \"a194673e-27c9-4ae9-bc0b-868a812a0b12\" (UID: \"a194673e-27c9-4ae9-bc0b-868a812a0b12\") "
Sep 16 19:12:51.944411 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:51.944387    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a194673e-27c9-4ae9-bc0b-868a812a0b12-kube-api-access-scdc4" (OuterVolumeSpecName: "kube-api-access-scdc4") pod "a194673e-27c9-4ae9-bc0b-868a812a0b12" (UID: "a194673e-27c9-4ae9-bc0b-868a812a0b12"). InnerVolumeSpecName "kube-api-access-scdc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:51.963124 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:51.963102    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:52.043243 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:52.043208    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scdc4\" (UniqueName: \"kubernetes.io/projected/a194673e-27c9-4ae9-bc0b-868a812a0b12-kube-api-access-scdc4\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:12:52.222030 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:52.221996    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a194673e-27c9-4ae9-bc0b-868a812a0b12" path="/var/lib/kubelet/pods/a194673e-27c9-4ae9-bc0b-868a812a0b12/volumes"
Sep 16 19:12:52.666205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:52.666116    2559 generic.go:356] "Generic (PLEG): container finished" podID="a194673e-27c9-4ae9-bc0b-868a812a0b12" containerID="77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704" exitCode=137
Sep 16 19:12:52.666205 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:52.666159    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-433364"
Sep 16 19:12:52.666434 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:52.666240    2559 scope.go:117] "RemoveContainer" containerID="77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704"
Sep 16 19:12:52.677375 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:52.677357    2559 scope.go:117] "RemoveContainer" containerID="77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704"
Sep 16 19:12:52.677632 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:12:52.677614    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704\": container with ID starting with 77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704 not found: ID does not exist" containerID="77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704"
Sep 16 19:12:52.677722 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:52.677638    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704"} err="failed to get container status \"77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704\": rpc error: code = NotFound desc = could not find container \"77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704\": container with ID starting with 77f619f7c78c09dbdd91e449cf37b10fd1270fd6497f6899bf3e149534768704 not found: ID does not exist"
Sep 16 19:12:53.849133 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:53.849092    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:54.138633 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:54.138556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:54.221651 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:12:54.221616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:04.299583 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.299547    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"]
Sep 16 19:13:04.300071 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.299987    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="376ea665-248b-42c0-b4e4-9c4864d5d3b1" containerName="main"
Sep 16 19:13:04.300071 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.300007    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="376ea665-248b-42c0-b4e4-9c4864d5d3b1" containerName="main"
Sep 16 19:13:04.300071 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.300025    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a194673e-27c9-4ae9-bc0b-868a812a0b12" containerName="test-curl-7226-433364"
Sep 16 19:13:04.300071 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.300033    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a194673e-27c9-4ae9-bc0b-868a812a0b12" containerName="test-curl-7226-433364"
Sep 16 19:13:04.300071 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.300052    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7099c17b-763b-4c64-b8af-3c64de776e64" containerName="main"
Sep 16 19:13:04.300071 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.300061    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7099c17b-763b-4c64-b8af-3c64de776e64" containerName="main"
Sep 16 19:13:04.300410 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.300151    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7099c17b-763b-4c64-b8af-3c64de776e64" containerName="main"
Sep 16 19:13:04.300410 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.300165    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a194673e-27c9-4ae9-bc0b-868a812a0b12" containerName="test-curl-7226-433364"
Sep 16 19:13:04.300410 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.300180    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="376ea665-248b-42c0-b4e4-9c4864d5d3b1" containerName="main"
Sep 16 19:13:04.303080 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.303059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.307764 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.307736    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca3f9a\"/\"sub-test-model-ca3f9a-kserve-self-signed-certs\""
Sep 16 19:13:04.308783 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.308761    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca3f9a\"/\"kube-root-ca.crt\""
Sep 16 19:13:04.309089 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.309055    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca3f9a\"/\"openshift-service-ca.crt\""
Sep 16 19:13:04.309208 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.309128    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca3f9a\"/\"default-dockercfg-wfs2j\""
Sep 16 19:13:04.325333 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.325303    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"]
Sep 16 19:13:04.330028 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.329998    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xggxw\" (UniqueName: \"kubernetes.io/projected/c47acb33-c0e2-48af-a65b-08dcd781c13d-kube-api-access-xggxw\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.330188 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.330043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-tmp-dir\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.330188 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.330099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47acb33-c0e2-48af-a65b-08dcd781c13d-tls-certs\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.330188 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.330119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-home\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.330188 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.330147    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-dshm\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.330188 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.330166    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-model-cache\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.333288 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.333263    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:04.430687 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.430649    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-model-cache\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.430878 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.430713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xggxw\" (UniqueName: \"kubernetes.io/projected/c47acb33-c0e2-48af-a65b-08dcd781c13d-kube-api-access-xggxw\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.430878 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.430738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-tmp-dir\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.430878 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.430769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47acb33-c0e2-48af-a65b-08dcd781c13d-tls-certs\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.431099 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.430907    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-home\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.431099 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.430957    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-dshm\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.431206 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.431157    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-tmp-dir\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.431206 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.431182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-model-cache\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.431317 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.431233    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-home\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.433178 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.433156    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-dshm\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.433178 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.433170    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47acb33-c0e2-48af-a65b-08dcd781c13d-tls-certs\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.440592 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.440572    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xggxw\" (UniqueName: \"kubernetes.io/projected/c47acb33-c0e2-48af-a65b-08dcd781c13d-kube-api-access-xggxw\") pod \"sub-test-model-ca3f9a-kserve-6659b74ffc-72796\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.612477 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.612390    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:04.886568 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:04.886542    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"]
Sep 16 19:13:04.888766 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:13:04.888730    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc47acb33_c0e2_48af_a65b_08dcd781c13d.slice/crio-37f821d05ca03942095090a95616ed3f1d39d450bcdbeb4cc8262d3423595407 WatchSource:0}: Error finding container 37f821d05ca03942095090a95616ed3f1d39d450bcdbeb4cc8262d3423595407: Status 404 returned error can't find the container with id 37f821d05ca03942095090a95616ed3f1d39d450bcdbeb4cc8262d3423595407
Sep 16 19:13:05.725069 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:05.725030    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796" event={"ID":"c47acb33-c0e2-48af-a65b-08dcd781c13d","Type":"ContainerStarted","Data":"fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47"}
Sep 16 19:13:05.725463 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:05.725079    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796" event={"ID":"c47acb33-c0e2-48af-a65b-08dcd781c13d","Type":"ContainerStarted","Data":"37f821d05ca03942095090a95616ed3f1d39d450bcdbeb4cc8262d3423595407"}
Sep 16 19:13:05.746680 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:05.746619    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796" podStartSLOduration=1.746603436 podStartE2EDuration="1.746603436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:05.744389567 +0000 UTC m=+2018.112445485" watchObservedRunningTime="2026-09-16 19:13:05.746603436 +0000 UTC m=+2018.114659351"
Sep 16 19:13:11.818127 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:11.818099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:11.921805 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:11.921770    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:12.000562 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:12.000534    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:14.613459 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:14.613418    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:14.613837 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:14.613472    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:14.627107 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:14.627081    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:14.776748 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:14.776716    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:13:24.912980 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:24.912944    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:24.983778 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:24.983731    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:27.949065 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:27.949027    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:28.322277 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:28.322246    2559 scope.go:117] "RemoveContainer" containerID="1b56bd4e3e210fbd68e55d2e2f0abc65bbe0bf38cfc8a84a96cb53e8e4ae1ac0"
Sep 16 19:13:36.684115 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.684082    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"]
Sep 16 19:13:36.687427 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.687403    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.690442 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.690413    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2dc9fc\"/\"default-dockercfg-g7tk4\""
Sep 16 19:13:36.690442 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.690426    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2dc9fc\"/\"openshift-service-ca.crt\""
Sep 16 19:13:36.691774 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.691757    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2dc9fc\"/\"sub-test-model-2dc9fc-kserve-self-signed-certs\""
Sep 16 19:13:36.691885 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.691868    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2dc9fc\"/\"kube-root-ca.crt\""
Sep 16 19:13:36.699632 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.699612    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"]
Sep 16 19:13:36.708995 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.708960    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:36.796430 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.796395    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-dshm\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.796585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.796436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589zl\" (UniqueName: \"kubernetes.io/projected/923a26f7-2049-4ffb-819c-d0e1190bb2e8-kube-api-access-589zl\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.796585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.796467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-home\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.796585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.796496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tmp-dir\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.796585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.796514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-model-cache\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.796585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.796535    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tls-certs\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.897375 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897341    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-model-cache\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tls-certs\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-dshm\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589zl\" (UniqueName: \"kubernetes.io/projected/923a26f7-2049-4ffb-819c-d0e1190bb2e8-kube-api-access-589zl\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897476    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-home\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tmp-dir\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.897815 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-model-cache\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.897891 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897861    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tmp-dir\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.898001 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.897895    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-home\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.899679 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.899656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-dshm\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.899846 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.899830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tls-certs\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.907379 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.907358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-589zl\" (UniqueName: \"kubernetes.io/projected/923a26f7-2049-4ffb-819c-d0e1190bb2e8-kube-api-access-589zl\") pod \"sub-test-model-2dc9fc-kserve-769dc789fc-kvs29\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:36.996509 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:36.996469    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:37.278236 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:37.278209    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"]
Sep 16 19:13:37.280685 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:13:37.280653    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod923a26f7_2049_4ffb_819c_d0e1190bb2e8.slice/crio-d62299fd682693789f176589f94ca6def46d09ae874af766c56d02d69236113f WatchSource:0}: Error finding container d62299fd682693789f176589f94ca6def46d09ae874af766c56d02d69236113f: Status 404 returned error can't find the container with id d62299fd682693789f176589f94ca6def46d09ae874af766c56d02d69236113f
Sep 16 19:13:37.870073 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:37.870031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29" event={"ID":"923a26f7-2049-4ffb-819c-d0e1190bb2e8","Type":"ContainerStarted","Data":"c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71"}
Sep 16 19:13:37.870473 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:37.870081    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29" event={"ID":"923a26f7-2049-4ffb-819c-d0e1190bb2e8","Type":"ContainerStarted","Data":"d62299fd682693789f176589f94ca6def46d09ae874af766c56d02d69236113f"}
Sep 16 19:13:37.895076 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:37.895026    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29" podStartSLOduration=1.895010036 podStartE2EDuration="1.895010036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:37.894434574 +0000 UTC m=+2050.262490488" watchObservedRunningTime="2026-09-16 19:13:37.895010036 +0000 UTC m=+2050.263065938"
Sep 16 19:13:45.103652 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:45.103616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:46.688512 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:46.688481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:46.997207 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:46.997172    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:46.997495 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:46.997468    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:47.009720 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:47.009698    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:13:47.925139 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:13:47.925107    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:14:11.649846 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:11.649768    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7226-31607d"]
Sep 16 19:14:11.652940 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:11.652923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:11.656093 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:11.656065    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q2jrz\""
Sep 16 19:14:11.659499 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:11.659475    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7226-31607d"]
Sep 16 19:14:11.680600 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:11.680575    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vqn\" (UniqueName: \"kubernetes.io/projected/1c392ab1-6350-4f5e-9f95-7d55f630506e-kube-api-access-44vqn\") pod \"test-curl-7226-31607d\" (UID: \"1c392ab1-6350-4f5e-9f95-7d55f630506e\") " pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:11.782046 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:11.782011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vqn\" (UniqueName: \"kubernetes.io/projected/1c392ab1-6350-4f5e-9f95-7d55f630506e-kube-api-access-44vqn\") pod \"test-curl-7226-31607d\" (UID: \"1c392ab1-6350-4f5e-9f95-7d55f630506e\") " pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:11.791121 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:11.791096    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vqn\" (UniqueName: \"kubernetes.io/projected/1c392ab1-6350-4f5e-9f95-7d55f630506e-kube-api-access-44vqn\") pod \"test-curl-7226-31607d\" (UID: \"1c392ab1-6350-4f5e-9f95-7d55f630506e\") " pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:11.962917 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:11.962883    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:12.229597 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:12.229574    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7226-31607d"]
Sep 16 19:14:12.231913 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:14:12.231881    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c392ab1_6350_4f5e_9f95_7d55f630506e.slice/crio-5cce2bc17e21e1786da3ab191b30a631798f6f164fd2c3a8a314a84b3dcc9f73 WatchSource:0}: Error finding container 5cce2bc17e21e1786da3ab191b30a631798f6f164fd2c3a8a314a84b3dcc9f73: Status 404 returned error can't find the container with id 5cce2bc17e21e1786da3ab191b30a631798f6f164fd2c3a8a314a84b3dcc9f73
Sep 16 19:14:13.030430 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.030396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7226-31607d" event={"ID":"1c392ab1-6350-4f5e-9f95-7d55f630506e","Type":"ContainerStarted","Data":"0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f"}
Sep 16 19:14:13.030797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.030444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7226-31607d" event={"ID":"1c392ab1-6350-4f5e-9f95-7d55f630506e","Type":"ContainerStarted","Data":"5cce2bc17e21e1786da3ab191b30a631798f6f164fd2c3a8a314a84b3dcc9f73"}
Sep 16 19:14:13.047754 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.047700    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7226-31607d" podStartSLOduration=2.047683017 podStartE2EDuration="2.047683017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:13.046056166 +0000 UTC m=+2085.414112066" watchObservedRunningTime="2026-09-16 19:14:13.047683017 +0000 UTC m=+2085.415738915"
Sep 16 19:14:13.635951 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.635914    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7226-31607d"]
Sep 16 19:14:13.640257 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.640226    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7226-31607d"]
Sep 16 19:14:13.751255 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.751222    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7226-e55eb5"]
Sep 16 19:14:13.751548 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.751535    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c392ab1-6350-4f5e-9f95-7d55f630506e" containerName="test-curl-7226-31607d"
Sep 16 19:14:13.751610 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.751550    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c392ab1-6350-4f5e-9f95-7d55f630506e" containerName="test-curl-7226-31607d"
Sep 16 19:14:13.751648 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.751620    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c392ab1-6350-4f5e-9f95-7d55f630506e" containerName="test-curl-7226-31607d"
Sep 16 19:14:13.754432 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.754414    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-e55eb5"
Sep 16 19:14:13.763485 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.763451    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7226-e55eb5"]
Sep 16 19:14:13.797163 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.797131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42kjr\" (UniqueName: \"kubernetes.io/projected/ab7254c2-b4da-459d-bd73-473d72283633-kube-api-access-42kjr\") pod \"test-curl-7226-e55eb5\" (UID: \"ab7254c2-b4da-459d-bd73-473d72283633\") " pod="opendatahub/test-curl-7226-e55eb5"
Sep 16 19:14:13.897883 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.897851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42kjr\" (UniqueName: \"kubernetes.io/projected/ab7254c2-b4da-459d-bd73-473d72283633-kube-api-access-42kjr\") pod \"test-curl-7226-e55eb5\" (UID: \"ab7254c2-b4da-459d-bd73-473d72283633\") " pod="opendatahub/test-curl-7226-e55eb5"
Sep 16 19:14:13.906833 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:13.906807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42kjr\" (UniqueName: \"kubernetes.io/projected/ab7254c2-b4da-459d-bd73-473d72283633-kube-api-access-42kjr\") pod \"test-curl-7226-e55eb5\" (UID: \"ab7254c2-b4da-459d-bd73-473d72283633\") " pod="opendatahub/test-curl-7226-e55eb5"
Sep 16 19:14:14.034864 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:14.034785    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7226-31607d" container="test-curl-7226-31607d"
Sep 16 19:14:14.064030 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:14.064006    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-e55eb5"
Sep 16 19:14:14.332251 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:14.332228    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7226-e55eb5"]
Sep 16 19:14:14.334108 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:14:14.334083    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab7254c2_b4da_459d_bd73_473d72283633.slice/crio-ebcd4acd0c8912a04f528ca132cb2eb583668ed7b344ab5506a5164795191f96 WatchSource:0}: Error finding container ebcd4acd0c8912a04f528ca132cb2eb583668ed7b344ab5506a5164795191f96: Status 404 returned error can't find the container with id ebcd4acd0c8912a04f528ca132cb2eb583668ed7b344ab5506a5164795191f96
Sep 16 19:14:15.041015 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:15.040968    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7226-e55eb5" event={"ID":"ab7254c2-b4da-459d-bd73-473d72283633","Type":"ContainerStarted","Data":"39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83"}
Sep 16 19:14:15.041015 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:15.041021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7226-e55eb5" event={"ID":"ab7254c2-b4da-459d-bd73-473d72283633","Type":"ContainerStarted","Data":"ebcd4acd0c8912a04f528ca132cb2eb583668ed7b344ab5506a5164795191f96"}
Sep 16 19:14:15.041470 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:15.041067    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7226-31607d" podUID="1c392ab1-6350-4f5e-9f95-7d55f630506e" containerName="test-curl-7226-31607d" containerID="cri-o://0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f" gracePeriod=2
Sep 16 19:14:15.060802 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:15.060749    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7226-e55eb5" podStartSLOduration=2.060733986 podStartE2EDuration="2.060733986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:15.059241777 +0000 UTC m=+2087.427297679" watchObservedRunningTime="2026-09-16 19:14:15.060733986 +0000 UTC m=+2087.428789887"
Sep 16 19:14:15.645202 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:15.645165    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7226-e55eb5"]
Sep 16 19:14:15.649645 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:15.649604    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7226-e55eb5"]
Sep 16 19:14:16.045824 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:16.045792    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7226-e55eb5" container="test-curl-7226-e55eb5"
Sep 16 19:14:17.050561 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:17.050516    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7226-e55eb5" podUID="ab7254c2-b4da-459d-bd73-473d72283633" containerName="test-curl-7226-e55eb5" containerID="cri-o://39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83" gracePeriod=2
Sep 16 19:14:17.182016 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:17.181967    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:17.227512 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:17.227486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44vqn\" (UniqueName: \"kubernetes.io/projected/1c392ab1-6350-4f5e-9f95-7d55f630506e-kube-api-access-44vqn\") pod \"1c392ab1-6350-4f5e-9f95-7d55f630506e\" (UID: \"1c392ab1-6350-4f5e-9f95-7d55f630506e\") "
Sep 16 19:14:17.229457 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:17.229437    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c392ab1-6350-4f5e-9f95-7d55f630506e-kube-api-access-44vqn" (OuterVolumeSpecName: "kube-api-access-44vqn") pod "1c392ab1-6350-4f5e-9f95-7d55f630506e" (UID: "1c392ab1-6350-4f5e-9f95-7d55f630506e"). InnerVolumeSpecName "kube-api-access-44vqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:17.328775 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:17.328692    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44vqn\" (UniqueName: \"kubernetes.io/projected/1c392ab1-6350-4f5e-9f95-7d55f630506e-kube-api-access-44vqn\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:17.664094 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:17.664019    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"]
Sep 16 19:14:17.678179 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:17.678137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:18.056488 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:18.056453    2559 generic.go:356] "Generic (PLEG): container finished" podID="1c392ab1-6350-4f5e-9f95-7d55f630506e" containerID="0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f" exitCode=137
Sep 16 19:14:18.056901 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:18.056495    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:18.056901 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:18.056506    2559 scope.go:117] "RemoveContainer" containerID="0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f"
Sep 16 19:14:18.065605 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:18.065587    2559 scope.go:117] "RemoveContainer" containerID="0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f"
Sep 16 19:14:18.065853 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:18.065837    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f\": container with ID starting with 0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f not found: ID does not exist" containerID="0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f"
Sep 16 19:14:18.065913 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:18.065859    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f"} err="failed to get container status \"0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f\": rpc error: code = NotFound desc = could not find container \"0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f\": container with ID starting with 0c914db5e245c5802286475a25f37ea932bd7d73d1ed20bc3ef39e786f31fb4f not found: ID does not exist"
Sep 16 19:14:18.067668 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:18.067648    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7226-31607d\" is forbidden: User \"system:node:ip-10-0-141-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-134.ec2.internal' and this object" podUID="1c392ab1-6350-4f5e-9f95-7d55f630506e" pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:18.226392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:18.226360    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c392ab1-6350-4f5e-9f95-7d55f630506e" path="/var/lib/kubelet/pods/1c392ab1-6350-4f5e-9f95-7d55f630506e/volumes"
Sep 16 19:14:18.227251 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:18.227224    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7226-31607d\" is forbidden: User \"system:node:ip-10-0-141-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-134.ec2.internal' and this object" podUID="1c392ab1-6350-4f5e-9f95-7d55f630506e" pod="opendatahub/test-curl-7226-31607d"
Sep 16 19:14:18.576172 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:18.576138    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"]
Sep 16 19:14:18.600516 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:18.600482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:19.218124 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:19.218100    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-e55eb5"
Sep 16 19:14:19.243716 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:19.243684    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42kjr\" (UniqueName: \"kubernetes.io/projected/ab7254c2-b4da-459d-bd73-473d72283633-kube-api-access-42kjr\") pod \"ab7254c2-b4da-459d-bd73-473d72283633\" (UID: \"ab7254c2-b4da-459d-bd73-473d72283633\") "
Sep 16 19:14:19.245733 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:19.245713    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7254c2-b4da-459d-bd73-473d72283633-kube-api-access-42kjr" (OuterVolumeSpecName: "kube-api-access-42kjr") pod "ab7254c2-b4da-459d-bd73-473d72283633" (UID: "ab7254c2-b4da-459d-bd73-473d72283633"). InnerVolumeSpecName "kube-api-access-42kjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:19.345095 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:19.345002    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42kjr\" (UniqueName: \"kubernetes.io/projected/ab7254c2-b4da-459d-bd73-473d72283633-kube-api-access-42kjr\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:20.070349 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:20.070302    2559 generic.go:356] "Generic (PLEG): container finished" podID="ab7254c2-b4da-459d-bd73-473d72283633" containerID="39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83" exitCode=137
Sep 16 19:14:20.070552 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:20.070359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7226-e55eb5"
Sep 16 19:14:20.070552 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:20.070399    2559 scope.go:117] "RemoveContainer" containerID="39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83"
Sep 16 19:14:20.079287 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:20.079269    2559 scope.go:117] "RemoveContainer" containerID="39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83"
Sep 16 19:14:20.079595 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:20.079571    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83\": container with ID starting with 39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83 not found: ID does not exist" containerID="39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83"
Sep 16 19:14:20.079655 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:20.079604    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83"} err="failed to get container status \"39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83\": rpc error: code = NotFound desc = could not find container \"39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83\": container with ID starting with 39263d5901f6c76a7a82f93c5ed728501dea75e933ab29265019239f73433c83 not found: ID does not exist"
Sep 16 19:14:20.081749 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:20.081724    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7226-e55eb5\" is forbidden: User \"system:node:ip-10-0-141-134.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-134.ec2.internal' and this object" podUID="ab7254c2-b4da-459d-bd73-473d72283633" pod="opendatahub/test-curl-7226-e55eb5"
Sep 16 19:14:20.221858 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:20.221824    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7254c2-b4da-459d-bd73-473d72283633" path="/var/lib/kubelet/pods/ab7254c2-b4da-459d-bd73-473d72283633/volumes"
Sep 16 19:14:28.125513 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:28.125484    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:14:28.131362 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:28.131339    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:14:32.683199 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:32.683131    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796" podUID="c47acb33-c0e2-48af-a65b-08dcd781c13d" containerName="main" containerID="cri-o://fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47" gracePeriod=45
Sep 16 19:14:32.919040 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:32.919013    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:14:33.066440 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066403    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-dshm\") pod \"c47acb33-c0e2-48af-a65b-08dcd781c13d\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") "
Sep 16 19:14:33.066440 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066441    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-model-cache\") pod \"c47acb33-c0e2-48af-a65b-08dcd781c13d\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") "
Sep 16 19:14:33.066697 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066480    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47acb33-c0e2-48af-a65b-08dcd781c13d-tls-certs\") pod \"c47acb33-c0e2-48af-a65b-08dcd781c13d\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") "
Sep 16 19:14:33.066697 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066535    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-tmp-dir\") pod \"c47acb33-c0e2-48af-a65b-08dcd781c13d\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") "
Sep 16 19:14:33.066697 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066559    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-home\") pod \"c47acb33-c0e2-48af-a65b-08dcd781c13d\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") "
Sep 16 19:14:33.066697 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066603    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xggxw\" (UniqueName: \"kubernetes.io/projected/c47acb33-c0e2-48af-a65b-08dcd781c13d-kube-api-access-xggxw\") pod \"c47acb33-c0e2-48af-a65b-08dcd781c13d\" (UID: \"c47acb33-c0e2-48af-a65b-08dcd781c13d\") "
Sep 16 19:14:33.066910 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066813    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c47acb33-c0e2-48af-a65b-08dcd781c13d" (UID: "c47acb33-c0e2-48af-a65b-08dcd781c13d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.066910 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066831    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-home" (OuterVolumeSpecName: "home") pod "c47acb33-c0e2-48af-a65b-08dcd781c13d" (UID: "c47acb33-c0e2-48af-a65b-08dcd781c13d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.066910 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.066792    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c47acb33-c0e2-48af-a65b-08dcd781c13d" (UID: "c47acb33-c0e2-48af-a65b-08dcd781c13d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.068801 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.068771    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c47acb33-c0e2-48af-a65b-08dcd781c13d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c47acb33-c0e2-48af-a65b-08dcd781c13d" (UID: "c47acb33-c0e2-48af-a65b-08dcd781c13d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:33.068801 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.068769    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-dshm" (OuterVolumeSpecName: "dshm") pod "c47acb33-c0e2-48af-a65b-08dcd781c13d" (UID: "c47acb33-c0e2-48af-a65b-08dcd781c13d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.068953 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.068846    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47acb33-c0e2-48af-a65b-08dcd781c13d-kube-api-access-xggxw" (OuterVolumeSpecName: "kube-api-access-xggxw") pod "c47acb33-c0e2-48af-a65b-08dcd781c13d" (UID: "c47acb33-c0e2-48af-a65b-08dcd781c13d"). InnerVolumeSpecName "kube-api-access-xggxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:33.132021 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.131963    2559 generic.go:356] "Generic (PLEG): container finished" podID="c47acb33-c0e2-48af-a65b-08dcd781c13d" containerID="fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47" exitCode=0
Sep 16 19:14:33.132021 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.131999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796" event={"ID":"c47acb33-c0e2-48af-a65b-08dcd781c13d","Type":"ContainerDied","Data":"fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47"}
Sep 16 19:14:33.132021 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.132023    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"
Sep 16 19:14:33.132288 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.132046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796" event={"ID":"c47acb33-c0e2-48af-a65b-08dcd781c13d","Type":"ContainerDied","Data":"37f821d05ca03942095090a95616ed3f1d39d450bcdbeb4cc8262d3423595407"}
Sep 16 19:14:33.132288 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.132063    2559 scope.go:117] "RemoveContainer" containerID="fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47"
Sep 16 19:14:33.141153 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.141117    2559 scope.go:117] "RemoveContainer" containerID="fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47"
Sep 16 19:14:33.141373 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:33.141353    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47\": container with ID starting with fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47 not found: ID does not exist" containerID="fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47"
Sep 16 19:14:33.141430 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.141381    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47"} err="failed to get container status \"fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47\": rpc error: code = NotFound desc = could not find container \"fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47\": container with ID starting with fe8265b616395bfcff7f293c3590bc7367e6e3cd802b5b509d579c247e095e47 not found: ID does not exist"
Sep 16 19:14:33.156655 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.156633    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"]
Sep 16 19:14:33.161144 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.161122    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/sub-test-model-ca3f9a-kserve-6659b74ffc-72796"]
Sep 16 19:14:33.167986 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.167956    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xggxw\" (UniqueName: \"kubernetes.io/projected/c47acb33-c0e2-48af-a65b-08dcd781c13d-kube-api-access-xggxw\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.168044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.167997    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.168044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.168009    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.168044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.168019    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c47acb33-c0e2-48af-a65b-08dcd781c13d-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.168044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.168026    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.168044 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.168034    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c47acb33-c0e2-48af-a65b-08dcd781c13d-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.595301 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.595259    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29" podUID="923a26f7-2049-4ffb-819c-d0e1190bb2e8" containerName="main" containerID="cri-o://c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71" gracePeriod=45
Sep 16 19:14:33.843669 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.843643    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:14:33.977067 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977030    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tls-certs\") pod \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") "
Sep 16 19:14:33.977260 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977087    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-589zl\" (UniqueName: \"kubernetes.io/projected/923a26f7-2049-4ffb-819c-d0e1190bb2e8-kube-api-access-589zl\") pod \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") "
Sep 16 19:14:33.977260 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977117    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-dshm\") pod \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") "
Sep 16 19:14:33.977260 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977230    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-model-cache\") pod \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") "
Sep 16 19:14:33.977468 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977279    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tmp-dir\") pod \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") "
Sep 16 19:14:33.977468 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977338    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-home\") pod \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\" (UID: \"923a26f7-2049-4ffb-819c-d0e1190bb2e8\") "
Sep 16 19:14:33.977585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977500    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "923a26f7-2049-4ffb-819c-d0e1190bb2e8" (UID: "923a26f7-2049-4ffb-819c-d0e1190bb2e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.977585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977517    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "923a26f7-2049-4ffb-819c-d0e1190bb2e8" (UID: "923a26f7-2049-4ffb-819c-d0e1190bb2e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.977696 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977609    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.977696 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977621    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:33.977696 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.977648    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-home" (OuterVolumeSpecName: "home") pod "923a26f7-2049-4ffb-819c-d0e1190bb2e8" (UID: "923a26f7-2049-4ffb-819c-d0e1190bb2e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.979372 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.979344    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-dshm" (OuterVolumeSpecName: "dshm") pod "923a26f7-2049-4ffb-819c-d0e1190bb2e8" (UID: "923a26f7-2049-4ffb-819c-d0e1190bb2e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:33.979486 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.979376    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "923a26f7-2049-4ffb-819c-d0e1190bb2e8" (UID: "923a26f7-2049-4ffb-819c-d0e1190bb2e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:33.979486 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:33.979472    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923a26f7-2049-4ffb-819c-d0e1190bb2e8-kube-api-access-589zl" (OuterVolumeSpecName: "kube-api-access-589zl") pod "923a26f7-2049-4ffb-819c-d0e1190bb2e8" (UID: "923a26f7-2049-4ffb-819c-d0e1190bb2e8"). InnerVolumeSpecName "kube-api-access-589zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:34.079077 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.079029    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:34.079077 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.079068    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/923a26f7-2049-4ffb-819c-d0e1190bb2e8-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:34.079077 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.079081    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589zl\" (UniqueName: \"kubernetes.io/projected/923a26f7-2049-4ffb-819c-d0e1190bb2e8-kube-api-access-589zl\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:34.079320 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.079090    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/923a26f7-2049-4ffb-819c-d0e1190bb2e8-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:14:34.137867 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.137834    2559 generic.go:356] "Generic (PLEG): container finished" podID="923a26f7-2049-4ffb-819c-d0e1190bb2e8" containerID="c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71" exitCode=0
Sep 16 19:14:34.138057 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.137874    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"
Sep 16 19:14:34.138057 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.137923    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29" event={"ID":"923a26f7-2049-4ffb-819c-d0e1190bb2e8","Type":"ContainerDied","Data":"c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71"}
Sep 16 19:14:34.138057 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.137984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29" event={"ID":"923a26f7-2049-4ffb-819c-d0e1190bb2e8","Type":"ContainerDied","Data":"d62299fd682693789f176589f94ca6def46d09ae874af766c56d02d69236113f"}
Sep 16 19:14:34.138057 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.138007    2559 scope.go:117] "RemoveContainer" containerID="c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71"
Sep 16 19:14:34.146896 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.146870    2559 scope.go:117] "RemoveContainer" containerID="c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71"
Sep 16 19:14:34.147208 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:14:34.147188    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71\": container with ID starting with c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71 not found: ID does not exist" containerID="c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71"
Sep 16 19:14:34.147272 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.147218    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71"} err="failed to get container status \"c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71\": rpc error: code = NotFound desc = could not find container \"c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71\": container with ID starting with c5da4f48245145f5d716892619fa589077b641fb7b4892574b0e931a73c04f71 not found: ID does not exist"
Sep 16 19:14:34.159164 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.159137    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"]
Sep 16 19:14:34.164248 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.164222    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/sub-test-model-2dc9fc-kserve-769dc789fc-kvs29"]
Sep 16 19:14:34.226797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.223388    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="923a26f7-2049-4ffb-819c-d0e1190bb2e8" path="/var/lib/kubelet/pods/923a26f7-2049-4ffb-819c-d0e1190bb2e8/volumes"
Sep 16 19:14:34.226797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:34.223957    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47acb33-c0e2-48af-a65b-08dcd781c13d" path="/var/lib/kubelet/pods/c47acb33-c0e2-48af-a65b-08dcd781c13d/volumes"
Sep 16 19:14:37.147638 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.147606    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"]
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.147892    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47acb33-c0e2-48af-a65b-08dcd781c13d" containerName="main"
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.147907    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47acb33-c0e2-48af-a65b-08dcd781c13d" containerName="main"
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.147935    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7254c2-b4da-459d-bd73-473d72283633" containerName="test-curl-7226-e55eb5"
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.147943    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7254c2-b4da-459d-bd73-473d72283633" containerName="test-curl-7226-e55eb5"
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.147956    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="923a26f7-2049-4ffb-819c-d0e1190bb2e8" containerName="main"
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.147964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="923a26f7-2049-4ffb-819c-d0e1190bb2e8" containerName="main"
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.148046    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47acb33-c0e2-48af-a65b-08dcd781c13d" containerName="main"
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.148054    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7254c2-b4da-459d-bd73-473d72283633" containerName="test-curl-7226-e55eb5"
Sep 16 19:14:37.148068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.148062    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="923a26f7-2049-4ffb-819c-d0e1190bb2e8" containerName="main"
Sep 16 19:14:37.152147 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.152120    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.157216 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.157186    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca3f9a\"/\"openshift-service-ca.crt\""
Sep 16 19:14:37.157800 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.157775    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca3f9a\"/\"default-dockercfg-wfs2j\""
Sep 16 19:14:37.158717 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.158674    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca3f9a\"/\"kube-root-ca.crt\""
Sep 16 19:14:37.158717 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.158701    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ca3f9a\"/\"rate-test-model-ca3f9a-kserve-self-signed-certs\""
Sep 16 19:14:37.177420 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.177395    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"]
Sep 16 19:14:37.180498 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.180475    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:37.303640 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.303600    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-tmp-dir\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.303640 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.303644    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-dshm\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.303873 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.303665    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-home\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.303873 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.303724    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed90bd0-7dfb-4d43-a077-21a06184f613-tls-certs\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.303873 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.303827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-model-cache\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.303873 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.303861    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njnj\" (UniqueName: \"kubernetes.io/projected/bed90bd0-7dfb-4d43-a077-21a06184f613-kube-api-access-4njnj\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.404813 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.404720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-model-cache\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.404813 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.404771    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njnj\" (UniqueName: \"kubernetes.io/projected/bed90bd0-7dfb-4d43-a077-21a06184f613-kube-api-access-4njnj\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.405075 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.404819    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-tmp-dir\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.405075 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.404851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-dshm\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.405075 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.404878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-home\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.405075 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.404908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed90bd0-7dfb-4d43-a077-21a06184f613-tls-certs\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.405281 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.405250    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-home\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.405281 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.405247    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-model-cache\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.405364 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.405327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-tmp-dir\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.407252 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.407228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-dshm\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.407382 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.407365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed90bd0-7dfb-4d43-a077-21a06184f613-tls-certs\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.418671 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.418644    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njnj\" (UniqueName: \"kubernetes.io/projected/bed90bd0-7dfb-4d43-a077-21a06184f613-kube-api-access-4njnj\") pod \"rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") " pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.460862 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.460828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:37.737033 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:37.737005    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"]
Sep 16 19:14:37.738806 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:14:37.738780    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbed90bd0_7dfb_4d43_a077_21a06184f613.slice/crio-5c1984a90b76de26da25af7c2af7a8d3c9157097c4c9f3bbe9e874f747555257 WatchSource:0}: Error finding container 5c1984a90b76de26da25af7c2af7a8d3c9157097c4c9f3bbe9e874f747555257: Status 404 returned error can't find the container with id 5c1984a90b76de26da25af7c2af7a8d3c9157097c4c9f3bbe9e874f747555257
Sep 16 19:14:38.158650 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:38.158561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj" event={"ID":"bed90bd0-7dfb-4d43-a077-21a06184f613","Type":"ContainerStarted","Data":"a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0"}
Sep 16 19:14:38.158650 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:38.158601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj" event={"ID":"bed90bd0-7dfb-4d43-a077-21a06184f613","Type":"ContainerStarted","Data":"5c1984a90b76de26da25af7c2af7a8d3c9157097c4c9f3bbe9e874f747555257"}
Sep 16 19:14:38.182145 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:38.182088    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj" podStartSLOduration=1.182071532 podStartE2EDuration="1.182071532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:38.180365088 +0000 UTC m=+2110.548420988" watchObservedRunningTime="2026-09-16 19:14:38.182071532 +0000 UTC m=+2110.550127432"
Sep 16 19:14:47.461846 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:47.461796    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:47.461846 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:47.461853    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:47.474562 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:47.474538    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:14:48.214060 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:14:48.214027    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:15:00.190450 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.190411    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"]
Sep 16 19:15:00.193697 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.193667    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"
Sep 16 19:15:00.203165 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.203145    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:15:00.209192 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.209171    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"]
Sep 16 19:15:00.212550 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.212530    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"
Sep 16 19:15:00.213085 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.213064    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"
Sep 16 19:15:00.234184 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.234160    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"]
Sep 16 19:15:00.244025 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.244000    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"
Sep 16 19:15:00.258999 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.258950    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"]
Sep 16 19:15:00.494292 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.494265    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"]
Sep 16 19:15:00.496677 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:15:00.496652    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d9787d_a19a_426c_87c5_cf20f192e9b2.slice/crio-bae49cedf7d20af50b61e41658124dcaa30a19d0ae3065a4d239b379c55cee75 WatchSource:0}: Error finding container bae49cedf7d20af50b61e41658124dcaa30a19d0ae3065a4d239b379c55cee75: Status 404 returned error can't find the container with id bae49cedf7d20af50b61e41658124dcaa30a19d0ae3065a4d239b379c55cee75
Sep 16 19:15:00.527870 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:00.527844    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"]
Sep 16 19:15:00.529779 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:15:00.529753    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb8c646_53fc_462c_8146_476db2e5118d.slice/crio-337f246e7130aab931444d16ce124f5b5e6973c2631c7dcaf270095542a374fe WatchSource:0}: Error finding container 337f246e7130aab931444d16ce124f5b5e6973c2631c7dcaf270095542a374fe: Status 404 returned error can't find the container with id 337f246e7130aab931444d16ce124f5b5e6973c2631c7dcaf270095542a374fe
Sep 16 19:15:01.086153 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.086068    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"]
Sep 16 19:15:01.089765 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.089743    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.094185 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.094165    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2dc9fc\"/\"default-dockercfg-g7tk4\""
Sep 16 19:15:01.095432 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.095406    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2dc9fc\"/\"kube-root-ca.crt\""
Sep 16 19:15:01.095580 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.095563    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-2dc9fc\"/\"rate-test-model-2dc9fc-kserve-self-signed-certs\""
Sep 16 19:15:01.095673 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.095583    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-2dc9fc\"/\"openshift-service-ca.crt\""
Sep 16 19:15:01.104268 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.104246    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"]
Sep 16 19:15:01.120263 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.120237    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:01.193557 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.193520    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-home\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.193557 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.193559    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49aef0d-bf25-44e6-ad79-36664c261a8c-tls-certs\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.194009 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.193591    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-dshm\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.194009 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.193661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-model-cache\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.194009 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.193735    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwd7\" (UniqueName: \"kubernetes.io/projected/a49aef0d-bf25-44e6-ad79-36664c261a8c-kube-api-access-xxwd7\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.194009 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.193789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-tmp-dir\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.260201 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.260167    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7" event={"ID":"d2d9787d-a19a-426c-87c5-cf20f192e9b2","Type":"ContainerStarted","Data":"212fc8bda5e8f192faa577ff71a368f4b4f7e1ab273744b7345ba56cb5566eef"}
Sep 16 19:15:01.260397 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.260223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7" event={"ID":"d2d9787d-a19a-426c-87c5-cf20f192e9b2","Type":"ContainerStarted","Data":"bae49cedf7d20af50b61e41658124dcaa30a19d0ae3065a4d239b379c55cee75"}
Sep 16 19:15:01.261675 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.261653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9" event={"ID":"fcb8c646-53fc-462c-8146-476db2e5118d","Type":"ContainerStarted","Data":"702388d8d477757f1cdb3884cc4f6184e9c96232d575df396ae89faa38fb49ae"}
Sep 16 19:15:01.261781 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.261693    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9" event={"ID":"fcb8c646-53fc-462c-8146-476db2e5118d","Type":"ContainerStarted","Data":"337f246e7130aab931444d16ce124f5b5e6973c2631c7dcaf270095542a374fe"}
Sep 16 19:15:01.279928 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.279884    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7" podStartSLOduration=1.279869475 podStartE2EDuration="1.279869475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:01.278385779 +0000 UTC m=+2133.646441701" watchObservedRunningTime="2026-09-16 19:15:01.279869475 +0000 UTC m=+2133.647925376"
Sep 16 19:15:01.295107 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295073    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-model-cache\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.295299 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwd7\" (UniqueName: \"kubernetes.io/projected/a49aef0d-bf25-44e6-ad79-36664c261a8c-kube-api-access-xxwd7\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.295299 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-tmp-dir\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.295299 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-home\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.295299 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49aef0d-bf25-44e6-ad79-36664c261a8c-tls-certs\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.295761 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-dshm\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.295862 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-model-cache\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.295997 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-tmp-dir\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.295997 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.295993    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-home\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.296703 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.296664    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9" podStartSLOduration=1.296651868 podStartE2EDuration="1.296651868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:01.296133092 +0000 UTC m=+2133.664189022" watchObservedRunningTime="2026-09-16 19:15:01.296651868 +0000 UTC m=+2133.664707769"
Sep 16 19:15:01.297772 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.297751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-dshm\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.298199 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.298182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49aef0d-bf25-44e6-ad79-36664c261a8c-tls-certs\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.304893 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.304868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwd7\" (UniqueName: \"kubernetes.io/projected/a49aef0d-bf25-44e6-ad79-36664c261a8c-kube-api-access-xxwd7\") pod \"rate-test-model-2dc9fc-kserve-86644844cf-klb52\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") " pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.398610 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.398578    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:01.671098 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:01.671007    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"]
Sep 16 19:15:01.674690 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:15:01.674662    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda49aef0d_bf25_44e6_ad79_36664c261a8c.slice/crio-722228336b683b93cbd3985f87205337c8908349f0341480d4afff90b8066364 WatchSource:0}: Error finding container 722228336b683b93cbd3985f87205337c8908349f0341480d4afff90b8066364: Status 404 returned error can't find the container with id 722228336b683b93cbd3985f87205337c8908349f0341480d4afff90b8066364
Sep 16 19:15:02.267911 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:02.267877    2559 generic.go:356] "Generic (PLEG): container finished" podID="fcb8c646-53fc-462c-8146-476db2e5118d" containerID="702388d8d477757f1cdb3884cc4f6184e9c96232d575df396ae89faa38fb49ae" exitCode=0
Sep 16 19:15:02.268357 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:02.267987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9" event={"ID":"fcb8c646-53fc-462c-8146-476db2e5118d","Type":"ContainerDied","Data":"702388d8d477757f1cdb3884cc4f6184e9c96232d575df396ae89faa38fb49ae"}
Sep 16 19:15:02.269511 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:02.269491    2559 generic.go:356] "Generic (PLEG): container finished" podID="d2d9787d-a19a-426c-87c5-cf20f192e9b2" containerID="212fc8bda5e8f192faa577ff71a368f4b4f7e1ab273744b7345ba56cb5566eef" exitCode=0
Sep 16 19:15:02.269576 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:02.269561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7" event={"ID":"d2d9787d-a19a-426c-87c5-cf20f192e9b2","Type":"ContainerDied","Data":"212fc8bda5e8f192faa577ff71a368f4b4f7e1ab273744b7345ba56cb5566eef"}
Sep 16 19:15:02.270935 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:02.270915    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52" event={"ID":"a49aef0d-bf25-44e6-ad79-36664c261a8c","Type":"ContainerStarted","Data":"78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08"}
Sep 16 19:15:02.271030 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:02.270944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52" event={"ID":"a49aef0d-bf25-44e6-ad79-36664c261a8c","Type":"ContainerStarted","Data":"722228336b683b93cbd3985f87205337c8908349f0341480d4afff90b8066364"}
Sep 16 19:15:02.319042 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:02.318962    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52" podStartSLOduration=1.3189458649999999 podStartE2EDuration="1.318945865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:02.317102013 +0000 UTC m=+2134.685157914" watchObservedRunningTime="2026-09-16 19:15:02.318945865 +0000 UTC m=+2134.687001766"
Sep 16 19:15:03.417258 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:03.417236    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"
Sep 16 19:15:03.423920 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:03.423900    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"
Sep 16 19:15:04.282965 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:04.282932    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"
Sep 16 19:15:04.283177 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:04.282965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9" event={"ID":"fcb8c646-53fc-462c-8146-476db2e5118d","Type":"ContainerDied","Data":"337f246e7130aab931444d16ce124f5b5e6973c2631c7dcaf270095542a374fe"}
Sep 16 19:15:04.283177 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:04.283037    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="337f246e7130aab931444d16ce124f5b5e6973c2631c7dcaf270095542a374fe"
Sep 16 19:15:04.284241 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:04.284224    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"
Sep 16 19:15:04.284374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:04.284251    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7" event={"ID":"d2d9787d-a19a-426c-87c5-cf20f192e9b2","Type":"ContainerDied","Data":"bae49cedf7d20af50b61e41658124dcaa30a19d0ae3065a4d239b379c55cee75"}
Sep 16 19:15:04.284374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:04.284289    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bae49cedf7d20af50b61e41658124dcaa30a19d0ae3065a4d239b379c55cee75"
Sep 16 19:15:11.399807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:11.399773    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:11.400339 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:11.399828    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:11.412250 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:11.412226    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:12.331451 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:12.331407    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:28.210055 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:28.210021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:37.497635 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:37.497601    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"]
Sep 16 19:15:37.516072 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:37.516043    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:38.469083 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:38.469047    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"]
Sep 16 19:15:38.497148 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:38.497118    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:38.924447 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:38.924351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:39.278303 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:39.278262    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:39.486188 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:39.486156    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:42.154718 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:42.154677    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:42.967497 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:42.967465    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:43.066445 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:43.066414    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:43.180935 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:43.180904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:44.315244 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:44.315162    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:45.104823 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.104794    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:45.204770 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.204739    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:45.519486 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.519455    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:45.623313 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.623280    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:45.719739 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.719705    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:45.771862 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.771788    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"]
Sep 16 19:15:45.772129 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.772115    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcb8c646-53fc-462c-8146-476db2e5118d" containerName="cleanup"
Sep 16 19:15:45.772181 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.772131    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcb8c646-53fc-462c-8146-476db2e5118d" containerName="cleanup"
Sep 16 19:15:45.772181 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.772142    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d9787d-a19a-426c-87c5-cf20f192e9b2" containerName="cleanup"
Sep 16 19:15:45.772181 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.772147    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d9787d-a19a-426c-87c5-cf20f192e9b2" containerName="cleanup"
Sep 16 19:15:45.772276 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.772203    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcb8c646-53fc-462c-8146-476db2e5118d" containerName="cleanup"
Sep 16 19:15:45.772276 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.772212    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d9787d-a19a-426c-87c5-cf20f192e9b2" containerName="cleanup"
Sep 16 19:15:45.775464 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.775448    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:45.778172 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.778151    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4992ef\""
Sep 16 19:15:45.778290 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.778267    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4992ef-dockercfg-x4f2l\""
Sep 16 19:15:45.783675 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.783652    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"]
Sep 16 19:15:45.816133 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.816106    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"]
Sep 16 19:15:45.819160 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.819144    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:45.829284 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.829259    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"]
Sep 16 19:15:45.873142 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.873108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07555e40-8c8f-48ea-9b6e-5bf102970a40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f\" (UID: \"07555e40-8c8f-48ea-9b6e-5bf102970a40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:45.873311 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.873149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntrb\" (UniqueName: \"kubernetes.io/projected/07555e40-8c8f-48ea-9b6e-5bf102970a40-kube-api-access-pntrb\") pod \"payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f\" (UID: \"07555e40-8c8f-48ea-9b6e-5bf102970a40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:45.973637 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.973589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a2a1a-3820-4e16-b631-ad5f1517036a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f\" (UID: \"256a2a1a-3820-4e16-b631-ad5f1517036a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:45.973848 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.973733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07555e40-8c8f-48ea-9b6e-5bf102970a40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f\" (UID: \"07555e40-8c8f-48ea-9b6e-5bf102970a40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:45.973848 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.973773    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4v82\" (UniqueName: \"kubernetes.io/projected/256a2a1a-3820-4e16-b631-ad5f1517036a-kube-api-access-l4v82\") pod \"payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f\" (UID: \"256a2a1a-3820-4e16-b631-ad5f1517036a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:45.973848 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.973797    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntrb\" (UniqueName: \"kubernetes.io/projected/07555e40-8c8f-48ea-9b6e-5bf102970a40-kube-api-access-pntrb\") pod \"payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f\" (UID: \"07555e40-8c8f-48ea-9b6e-5bf102970a40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:45.974503 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.974480    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07555e40-8c8f-48ea-9b6e-5bf102970a40-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f\" (UID: \"07555e40-8c8f-48ea-9b6e-5bf102970a40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:45.983461 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:45.983433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntrb\" (UniqueName: \"kubernetes.io/projected/07555e40-8c8f-48ea-9b6e-5bf102970a40-kube-api-access-pntrb\") pod \"payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f\" (UID: \"07555e40-8c8f-48ea-9b6e-5bf102970a40\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:46.074580 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.074497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4v82\" (UniqueName: \"kubernetes.io/projected/256a2a1a-3820-4e16-b631-ad5f1517036a-kube-api-access-l4v82\") pod \"payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f\" (UID: \"256a2a1a-3820-4e16-b631-ad5f1517036a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:46.074580 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.074543    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a2a1a-3820-4e16-b631-ad5f1517036a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f\" (UID: \"256a2a1a-3820-4e16-b631-ad5f1517036a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:46.075234 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.075215    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a2a1a-3820-4e16-b631-ad5f1517036a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f\" (UID: \"256a2a1a-3820-4e16-b631-ad5f1517036a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:46.084904 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.084880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4v82\" (UniqueName: \"kubernetes.io/projected/256a2a1a-3820-4e16-b631-ad5f1517036a-kube-api-access-l4v82\") pod \"payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f\" (UID: \"256a2a1a-3820-4e16-b631-ad5f1517036a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:46.086560 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.086534    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:46.128953 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.128920    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:46.356513 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.356487    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"]
Sep 16 19:15:46.358357 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:15:46.358330    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07555e40_8c8f_48ea_9b6e_5bf102970a40.slice/crio-dfd26d48b88911e76385e914c2a5ddb52951eeaa27cce52c4ff722f592405d75 WatchSource:0}: Error finding container dfd26d48b88911e76385e914c2a5ddb52951eeaa27cce52c4ff722f592405d75: Status 404 returned error can't find the container with id dfd26d48b88911e76385e914c2a5ddb52951eeaa27cce52c4ff722f592405d75
Sep 16 19:15:46.406674 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.406647    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"]
Sep 16 19:15:46.408137 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:15:46.408114    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256a2a1a_3820_4e16_b631_ad5f1517036a.slice/crio-3d4d9fe1a5755b11db36d4ad7ed48b9dccb165c0df017ac7640dfe629b8ab7bd WatchSource:0}: Error finding container 3d4d9fe1a5755b11db36d4ad7ed48b9dccb165c0df017ac7640dfe629b8ab7bd: Status 404 returned error can't find the container with id 3d4d9fe1a5755b11db36d4ad7ed48b9dccb165c0df017ac7640dfe629b8ab7bd
Sep 16 19:15:46.470765 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.470731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f" event={"ID":"256a2a1a-3820-4e16-b631-ad5f1517036a","Type":"ContainerStarted","Data":"d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a"}
Sep 16 19:15:46.470884 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.470792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f" event={"ID":"256a2a1a-3820-4e16-b631-ad5f1517036a","Type":"ContainerStarted","Data":"3d4d9fe1a5755b11db36d4ad7ed48b9dccb165c0df017ac7640dfe629b8ab7bd"}
Sep 16 19:15:46.471109 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.470912    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:15:46.472462 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.472444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f" event={"ID":"07555e40-8c8f-48ea-9b6e-5bf102970a40","Type":"ContainerStarted","Data":"9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80"}
Sep 16 19:15:46.472582 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.472489    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f" event={"ID":"07555e40-8c8f-48ea-9b6e-5bf102970a40","Type":"ContainerStarted","Data":"dfd26d48b88911e76385e914c2a5ddb52951eeaa27cce52c4ff722f592405d75"}
Sep 16 19:15:46.472582 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.472550    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:46.491305 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.491258    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f" podStartSLOduration=1.491245533 podStartE2EDuration="1.491245533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:46.489642653 +0000 UTC m=+2178.857698587" watchObservedRunningTime="2026-09-16 19:15:46.491245533 +0000 UTC m=+2178.859301433"
Sep 16 19:15:46.510136 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:46.510091    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f" podStartSLOduration=1.510079706 podStartE2EDuration="1.510079706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:46.507963689 +0000 UTC m=+2178.876019615" watchObservedRunningTime="2026-09-16 19:15:46.510079706 +0000 UTC m=+2178.878135608"
Sep 16 19:15:52.515994 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.515929    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj" podUID="bed90bd0-7dfb-4d43-a077-21a06184f613" containerName="main" containerID="cri-o://a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0" gracePeriod=45
Sep 16 19:15:52.757355 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.757330    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:15:52.934867 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.934786    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-model-cache\") pod \"bed90bd0-7dfb-4d43-a077-21a06184f613\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") "
Sep 16 19:15:52.934867 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.934834    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed90bd0-7dfb-4d43-a077-21a06184f613-tls-certs\") pod \"bed90bd0-7dfb-4d43-a077-21a06184f613\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") "
Sep 16 19:15:52.934867 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.934861    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-home\") pod \"bed90bd0-7dfb-4d43-a077-21a06184f613\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") "
Sep 16 19:15:52.935195 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.934900    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-dshm\") pod \"bed90bd0-7dfb-4d43-a077-21a06184f613\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") "
Sep 16 19:15:52.935195 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.935011    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-tmp-dir\") pod \"bed90bd0-7dfb-4d43-a077-21a06184f613\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") "
Sep 16 19:15:52.935195 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.935036    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4njnj\" (UniqueName: \"kubernetes.io/projected/bed90bd0-7dfb-4d43-a077-21a06184f613-kube-api-access-4njnj\") pod \"bed90bd0-7dfb-4d43-a077-21a06184f613\" (UID: \"bed90bd0-7dfb-4d43-a077-21a06184f613\") "
Sep 16 19:15:52.935195 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.935125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-model-cache" (OuterVolumeSpecName: "model-cache") pod "bed90bd0-7dfb-4d43-a077-21a06184f613" (UID: "bed90bd0-7dfb-4d43-a077-21a06184f613"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:52.935392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.935226    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-home" (OuterVolumeSpecName: "home") pod "bed90bd0-7dfb-4d43-a077-21a06184f613" (UID: "bed90bd0-7dfb-4d43-a077-21a06184f613"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:52.935392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.935265    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bed90bd0-7dfb-4d43-a077-21a06184f613" (UID: "bed90bd0-7dfb-4d43-a077-21a06184f613"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:52.935392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.935350    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:52.935392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.935368    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:52.935392 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.935379    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:52.937207 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.937181    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-dshm" (OuterVolumeSpecName: "dshm") pod "bed90bd0-7dfb-4d43-a077-21a06184f613" (UID: "bed90bd0-7dfb-4d43-a077-21a06184f613"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:52.937306 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.937213    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed90bd0-7dfb-4d43-a077-21a06184f613-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bed90bd0-7dfb-4d43-a077-21a06184f613" (UID: "bed90bd0-7dfb-4d43-a077-21a06184f613"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:52.937306 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:52.937246    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed90bd0-7dfb-4d43-a077-21a06184f613-kube-api-access-4njnj" (OuterVolumeSpecName: "kube-api-access-4njnj") pod "bed90bd0-7dfb-4d43-a077-21a06184f613" (UID: "bed90bd0-7dfb-4d43-a077-21a06184f613"). InnerVolumeSpecName "kube-api-access-4njnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:53.035865 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.035836    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njnj\" (UniqueName: \"kubernetes.io/projected/bed90bd0-7dfb-4d43-a077-21a06184f613-kube-api-access-4njnj\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.035865 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.035863    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed90bd0-7dfb-4d43-a077-21a06184f613-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.035865 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.035873    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed90bd0-7dfb-4d43-a077-21a06184f613-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.487579 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.487514    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52" podUID="a49aef0d-bf25-44e6-ad79-36664c261a8c" containerName="main" containerID="cri-o://78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08" gracePeriod=45
Sep 16 19:15:53.508666 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.508638    2559 generic.go:356] "Generic (PLEG): container finished" podID="bed90bd0-7dfb-4d43-a077-21a06184f613" containerID="a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0" exitCode=0
Sep 16 19:15:53.508797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.508683    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"
Sep 16 19:15:53.508797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.508716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj" event={"ID":"bed90bd0-7dfb-4d43-a077-21a06184f613","Type":"ContainerDied","Data":"a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0"}
Sep 16 19:15:53.508797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.508758    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj" event={"ID":"bed90bd0-7dfb-4d43-a077-21a06184f613","Type":"ContainerDied","Data":"5c1984a90b76de26da25af7c2af7a8d3c9157097c4c9f3bbe9e874f747555257"}
Sep 16 19:15:53.508797 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.508777    2559 scope.go:117] "RemoveContainer" containerID="a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0"
Sep 16 19:15:53.563966 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.563935    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"]
Sep 16 19:15:53.568820 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.568796    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ca3f9a/rate-test-model-ca3f9a-kserve-7c6f699c5-p2rhj"]
Sep 16 19:15:53.598700 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.598538    2559 scope.go:117] "RemoveContainer" containerID="a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0"
Sep 16 19:15:53.598861 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:15:53.598842    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0\": container with ID starting with a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0 not found: ID does not exist" containerID="a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0"
Sep 16 19:15:53.598929 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.598873    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0"} err="failed to get container status \"a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0\": rpc error: code = NotFound desc = could not find container \"a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0\": container with ID starting with a84870d0f4e6de95c2777396daf96fe8312727ba3526b756862bdcb04fb090b0 not found: ID does not exist"
Sep 16 19:15:53.734548 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.734524    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:53.841487 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841402    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxwd7\" (UniqueName: \"kubernetes.io/projected/a49aef0d-bf25-44e6-ad79-36664c261a8c-kube-api-access-xxwd7\") pod \"a49aef0d-bf25-44e6-ad79-36664c261a8c\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") "
Sep 16 19:15:53.841487 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841441    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49aef0d-bf25-44e6-ad79-36664c261a8c-tls-certs\") pod \"a49aef0d-bf25-44e6-ad79-36664c261a8c\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") "
Sep 16 19:15:53.841487 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841462    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-dshm\") pod \"a49aef0d-bf25-44e6-ad79-36664c261a8c\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") "
Sep 16 19:15:53.841487 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841478    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-model-cache\") pod \"a49aef0d-bf25-44e6-ad79-36664c261a8c\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") "
Sep 16 19:15:53.841822 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841576    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-home\") pod \"a49aef0d-bf25-44e6-ad79-36664c261a8c\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") "
Sep 16 19:15:53.841822 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841626    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-tmp-dir\") pod \"a49aef0d-bf25-44e6-ad79-36664c261a8c\" (UID: \"a49aef0d-bf25-44e6-ad79-36664c261a8c\") "
Sep 16 19:15:53.841822 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841784    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a49aef0d-bf25-44e6-ad79-36664c261a8c" (UID: "a49aef0d-bf25-44e6-ad79-36664c261a8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:53.842010 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841826    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-home" (OuterVolumeSpecName: "home") pod "a49aef0d-bf25-44e6-ad79-36664c261a8c" (UID: "a49aef0d-bf25-44e6-ad79-36664c261a8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:53.842010 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.841877    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a49aef0d-bf25-44e6-ad79-36664c261a8c" (UID: "a49aef0d-bf25-44e6-ad79-36664c261a8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:53.842010 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.842001    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.842130 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.842016    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.842130 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.842025    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.843798 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.843781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49aef0d-bf25-44e6-ad79-36664c261a8c-kube-api-access-xxwd7" (OuterVolumeSpecName: "kube-api-access-xxwd7") pod "a49aef0d-bf25-44e6-ad79-36664c261a8c" (UID: "a49aef0d-bf25-44e6-ad79-36664c261a8c"). InnerVolumeSpecName "kube-api-access-xxwd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:53.844084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.844063    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49aef0d-bf25-44e6-ad79-36664c261a8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a49aef0d-bf25-44e6-ad79-36664c261a8c" (UID: "a49aef0d-bf25-44e6-ad79-36664c261a8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:53.844084 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.844072    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-dshm" (OuterVolumeSpecName: "dshm") pod "a49aef0d-bf25-44e6-ad79-36664c261a8c" (UID: "a49aef0d-bf25-44e6-ad79-36664c261a8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:53.942803 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.942763    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxwd7\" (UniqueName: \"kubernetes.io/projected/a49aef0d-bf25-44e6-ad79-36664c261a8c-kube-api-access-xxwd7\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.942803 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.942798    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a49aef0d-bf25-44e6-ad79-36664c261a8c-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:53.942803 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:53.942806    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a49aef0d-bf25-44e6-ad79-36664c261a8c-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:15:54.222712 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.222667    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bed90bd0-7dfb-4d43-a077-21a06184f613" path="/var/lib/kubelet/pods/bed90bd0-7dfb-4d43-a077-21a06184f613/volumes"
Sep 16 19:15:54.514361 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.514261    2559 generic.go:356] "Generic (PLEG): container finished" podID="a49aef0d-bf25-44e6-ad79-36664c261a8c" containerID="78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08" exitCode=0
Sep 16 19:15:54.514361 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.514327    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"
Sep 16 19:15:54.514586 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.514353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52" event={"ID":"a49aef0d-bf25-44e6-ad79-36664c261a8c","Type":"ContainerDied","Data":"78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08"}
Sep 16 19:15:54.514586 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.514397    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52" event={"ID":"a49aef0d-bf25-44e6-ad79-36664c261a8c","Type":"ContainerDied","Data":"722228336b683b93cbd3985f87205337c8908349f0341480d4afff90b8066364"}
Sep 16 19:15:54.514586 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.514413    2559 scope.go:117] "RemoveContainer" containerID="78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08"
Sep 16 19:15:54.522728 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.522708    2559 scope.go:117] "RemoveContainer" containerID="78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08"
Sep 16 19:15:54.523007 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:15:54.522967    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08\": container with ID starting with 78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08 not found: ID does not exist" containerID="78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08"
Sep 16 19:15:54.523062 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.523016    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08"} err="failed to get container status \"78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08\": rpc error: code = NotFound desc = could not find container \"78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08\": container with ID starting with 78e05a0b77b8eea39ef1c141e6f4874c0936562725e09b50b863bf954807bf08 not found: ID does not exist"
Sep 16 19:15:54.532234 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.532215    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"]
Sep 16 19:15:54.537940 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:54.537918    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-2dc9fc/rate-test-model-2dc9fc-kserve-86644844cf-klb52"]
Sep 16 19:15:56.221950 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:56.221920    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49aef0d-bf25-44e6-ad79-36664c261a8c" path="/var/lib/kubelet/pods/a49aef0d-bf25-44e6-ad79-36664c261a8c/volumes"
Sep 16 19:15:57.478899 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:57.478867    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:15:57.479388 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:15:57.479055    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:16:08.529254 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:08.529223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:09.438058 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:09.438023    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:10.205462 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:10.205429    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:11.045426 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:11.045390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:12.899692 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:12.899659    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:13.557072 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:13.557037    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:13.856036 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:13.855950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:14.058363 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:14.058330    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:17.048134 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:17.048101    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:17.202210 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:17.202174    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:17.317913 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:17.317834    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:18.266035 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:18.266001    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:19.200799 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:19.200765    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:20.143655 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.143616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:20.238426 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.238390    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:20.392895 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.392858    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:20.714497 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.714462    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:20.805403 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.805374    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:20.928612 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.928577    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"]
Sep 16 19:16:20.928927 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.928915    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed90bd0-7dfb-4d43-a077-21a06184f613" containerName="main"
Sep 16 19:16:20.929007 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.928928    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed90bd0-7dfb-4d43-a077-21a06184f613" containerName="main"
Sep 16 19:16:20.929007 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.928939    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49aef0d-bf25-44e6-ad79-36664c261a8c" containerName="main"
Sep 16 19:16:20.929007 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.928945    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49aef0d-bf25-44e6-ad79-36664c261a8c" containerName="main"
Sep 16 19:16:20.929108 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.929020    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49aef0d-bf25-44e6-ad79-36664c261a8c" containerName="main"
Sep 16 19:16:20.929108 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.929030    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed90bd0-7dfb-4d43-a077-21a06184f613" containerName="main"
Sep 16 19:16:20.932115 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.932098    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:20.935414 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.935394    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-766919-dockercfg-729g8\""
Sep 16 19:16:20.935546 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.935465    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-766919\""
Sep 16 19:16:20.942332 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:20.942302    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"]
Sep 16 19:16:21.015013 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.014911    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"]
Sep 16 19:16:21.018246 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.018223    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:21.033821 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.033792    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"]
Sep 16 19:16:21.065556 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.065524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhz6p\" (UniqueName: \"kubernetes.io/projected/c61de80a-ace8-4f43-98ad-bd4eaf78c818-kube-api-access-nhz6p\") pod \"payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz\" (UID: \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:21.065758 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.065570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867e5c3a-d02a-4cea-9d22-e76c2973da40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h\" (UID: \"867e5c3a-d02a-4cea-9d22-e76c2973da40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:21.065758 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.065610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gc7j\" (UniqueName: \"kubernetes.io/projected/867e5c3a-d02a-4cea-9d22-e76c2973da40-kube-api-access-8gc7j\") pod \"payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h\" (UID: \"867e5c3a-d02a-4cea-9d22-e76c2973da40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:21.065758 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.065673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61de80a-ace8-4f43-98ad-bd4eaf78c818-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz\" (UID: \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:21.166844 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.166801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhz6p\" (UniqueName: \"kubernetes.io/projected/c61de80a-ace8-4f43-98ad-bd4eaf78c818-kube-api-access-nhz6p\") pod \"payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz\" (UID: \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:21.167293 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.166856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867e5c3a-d02a-4cea-9d22-e76c2973da40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h\" (UID: \"867e5c3a-d02a-4cea-9d22-e76c2973da40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:21.167293 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.166906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gc7j\" (UniqueName: \"kubernetes.io/projected/867e5c3a-d02a-4cea-9d22-e76c2973da40-kube-api-access-8gc7j\") pod \"payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h\" (UID: \"867e5c3a-d02a-4cea-9d22-e76c2973da40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:21.167293 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.166937    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61de80a-ace8-4f43-98ad-bd4eaf78c818-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz\" (UID: \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:21.167665 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.167643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867e5c3a-d02a-4cea-9d22-e76c2973da40-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h\" (UID: \"867e5c3a-d02a-4cea-9d22-e76c2973da40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:21.167703 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.167651    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61de80a-ace8-4f43-98ad-bd4eaf78c818-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz\" (UID: \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:21.176696 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.176671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gc7j\" (UniqueName: \"kubernetes.io/projected/867e5c3a-d02a-4cea-9d22-e76c2973da40-kube-api-access-8gc7j\") pod \"payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h\" (UID: \"867e5c3a-d02a-4cea-9d22-e76c2973da40\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:21.178124 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.178102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhz6p\" (UniqueName: \"kubernetes.io/projected/c61de80a-ace8-4f43-98ad-bd4eaf78c818-kube-api-access-nhz6p\") pod \"payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz\" (UID: \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:21.243593 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.243552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:21.329060 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.329025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:21.514267 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.514242    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"]
Sep 16 19:16:21.516708 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:16:21.516677    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867e5c3a_d02a_4cea_9d22_e76c2973da40.slice/crio-23e7d91a713e7c79b4924678ce023c553b346010547993341678ff117953f507 WatchSource:0}: Error finding container 23e7d91a713e7c79b4924678ce023c553b346010547993341678ff117953f507: Status 404 returned error can't find the container with id 23e7d91a713e7c79b4924678ce023c553b346010547993341678ff117953f507
Sep 16 19:16:21.602948 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.602923    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"]
Sep 16 19:16:21.604792 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:16:21.604766    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61de80a_ace8_4f43_98ad_bd4eaf78c818.slice/crio-7a43e62e6df3ec93b24f2bd2a898edad2b5c7f66ea7b4a3f8c7e6d1a56c73fa3 WatchSource:0}: Error finding container 7a43e62e6df3ec93b24f2bd2a898edad2b5c7f66ea7b4a3f8c7e6d1a56c73fa3: Status 404 returned error can't find the container with id 7a43e62e6df3ec93b24f2bd2a898edad2b5c7f66ea7b4a3f8c7e6d1a56c73fa3
Sep 16 19:16:21.645631 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.645601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz" event={"ID":"c61de80a-ace8-4f43-98ad-bd4eaf78c818","Type":"ContainerStarted","Data":"7a43e62e6df3ec93b24f2bd2a898edad2b5c7f66ea7b4a3f8c7e6d1a56c73fa3"}
Sep 16 19:16:21.647329 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.647307    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h" event={"ID":"867e5c3a-d02a-4cea-9d22-e76c2973da40","Type":"ContainerStarted","Data":"30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2"}
Sep 16 19:16:21.647426 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.647355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h" event={"ID":"867e5c3a-d02a-4cea-9d22-e76c2973da40","Type":"ContainerStarted","Data":"23e7d91a713e7c79b4924678ce023c553b346010547993341678ff117953f507"}
Sep 16 19:16:21.647426 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.647384    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:21.681208 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:21.681157    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h" podStartSLOduration=1.681139902 podStartE2EDuration="1.681139902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:21.679513448 +0000 UTC m=+2214.047569360" watchObservedRunningTime="2026-09-16 19:16:21.681139902 +0000 UTC m=+2214.049195794"
Sep 16 19:16:22.659180 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:22.659139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz" event={"ID":"c61de80a-ace8-4f43-98ad-bd4eaf78c818","Type":"ContainerStarted","Data":"3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6"}
Sep 16 19:16:22.681011 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:22.680945    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz" podStartSLOduration=2.680930275 podStartE2EDuration="2.680930275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:22.679594244 +0000 UTC m=+2215.047650156" watchObservedRunningTime="2026-09-16 19:16:22.680930275 +0000 UTC m=+2215.048986176"
Sep 16 19:16:23.665138 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:23.665104    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:24.207102 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:24.207070    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:32.661331 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:32.661301    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:16:34.673486 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:34.673447    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:16:41.333604 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:41.333570    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:42.816496 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:42.816465    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:43.858168 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:43.858136    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:45.017818 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:45.017784    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:45.901683 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:45.901650    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:46.621523 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:46.621487    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:47.181497 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:47.181463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:52.153420 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:52.153347    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w66f9_3f1972d0-318d-4e9a-b367-6745577a2e9a/payload-processing/0.log"
Sep 16 19:16:52.260928 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:52.260900    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz_c61de80a-ace8-4f43-98ad-bd4eaf78c818/payload-processing/0.log"
Sep 16 19:16:58.235415 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:16:58.235383    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:00.780630 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.780600    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"]
Sep 16 19:17:00.784232 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.784210    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.788759 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.788735    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4992ef\"/\"openshift-service-ca.crt\""
Sep 16 19:17:00.788893 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.788760    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4992ef\"/\"ipp-route-4992ef-kserve-self-signed-certs\""
Sep 16 19:17:00.788893 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.788738    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-4992ef\"/\"default-dockercfg-fbqfw\""
Sep 16 19:17:00.788893 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.788786    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-4992ef\"/\"kube-root-ca.crt\""
Sep 16 19:17:00.800724 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.800701    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"]
Sep 16 19:17:00.809161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.809137    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:00.897280 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.897246    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jzs\" (UniqueName: \"kubernetes.io/projected/7ce62def-233a-4304-b6c9-09caf79d59d5-kube-api-access-f2jzs\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.897280 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.897285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-home\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.897307    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-model-cache\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.897329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-tmp-dir\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.897407    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce62def-233a-4304-b6c9-09caf79d59d5-tls-certs\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.897542 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.897454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-dshm\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998168 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998132    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce62def-233a-4304-b6c9-09caf79d59d5-tls-certs\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998168 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-dshm\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jzs\" (UniqueName: \"kubernetes.io/projected/7ce62def-233a-4304-b6c9-09caf79d59d5-kube-api-access-f2jzs\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998267    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-home\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-model-cache\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998422 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-tmp-dir\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998736 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-tmp-dir\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998855 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998750    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-model-cache\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:00.998855 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:00.998777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-home\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:01.000503 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.000477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-dshm\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:01.000841 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.000824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce62def-233a-4304-b6c9-09caf79d59d5-tls-certs\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:01.007249 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.007215    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jzs\" (UniqueName: \"kubernetes.io/projected/7ce62def-233a-4304-b6c9-09caf79d59d5-kube-api-access-f2jzs\") pod \"ipp-route-4992ef-kserve-5d4896675-t78dp\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") " pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:01.093280 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.093195    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:01.366346 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.366314    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"]
Sep 16 19:17:01.368490 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:17:01.368457    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ce62def_233a_4304_b6c9_09caf79d59d5.slice/crio-d0a327b4a2a2566b0b29237e98e2c690cb64a233b7b7c97bf3defdda09a368eb WatchSource:0}: Error finding container d0a327b4a2a2566b0b29237e98e2c690cb64a233b7b7c97bf3defdda09a368eb: Status 404 returned error can't find the container with id d0a327b4a2a2566b0b29237e98e2c690cb64a233b7b7c97bf3defdda09a368eb
Sep 16 19:17:01.635145 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.635060    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:01.840882 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.840845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp" event={"ID":"7ce62def-233a-4304-b6c9-09caf79d59d5","Type":"ContainerStarted","Data":"5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729"}
Sep 16 19:17:01.840882 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.840887    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp" event={"ID":"7ce62def-233a-4304-b6c9-09caf79d59d5","Type":"ContainerStarted","Data":"d0a327b4a2a2566b0b29237e98e2c690cb64a233b7b7c97bf3defdda09a368eb"}
Sep 16 19:17:01.864106 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:01.864048    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp" podStartSLOduration=1.864032975 podStartE2EDuration="1.864032975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:01.863365544 +0000 UTC m=+2254.231421459" watchObservedRunningTime="2026-09-16 19:17:01.864032975 +0000 UTC m=+2254.232088875"
Sep 16 19:17:11.093892 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:11.093847    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:11.094335 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:11.093918    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:11.106415 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:11.106385    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:11.899638 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:11.899599    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:17:12.284024 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:12.283964    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:22.888433 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:22.888396    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f_07555e40-8c8f-48ea-9b6e-5bf102970a40/payload-processing/0.log"
Sep 16 19:17:23.008402 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:23.008373    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w66f9_3f1972d0-318d-4e9a-b367-6745577a2e9a/payload-processing/0.log"
Sep 16 19:17:23.134595 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:23.134569    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz_c61de80a-ace8-4f43-98ad-bd4eaf78c818/payload-processing/0.log"
Sep 16 19:17:23.473700 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:23.473664    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:23.817272 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:23.817201    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:24.124832 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:24.124747    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:25.864176 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:25.864142    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:26.501762 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:26.501725    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:26.592832 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:26.592791    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:26.827952 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:26.827868    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:27.988842 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:27.988811    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:28.397530 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:28.397453    2559 scope.go:117] "RemoveContainer" containerID="79977d88aeb9ffb40c93e5382aa7c986f5780afb4a1f330e055cf9a6c534ed73"
Sep 16 19:17:28.405644 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:28.405626    2559 scope.go:117] "RemoveContainer" containerID="245ab00e5db83d2239a028f18e95ac6f505881dada2b95bfd25f60494e827ca9"
Sep 16 19:17:28.924730 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:28.924700    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:29.728249 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:29.728220    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:30.637576 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:30.637547    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:30.706928 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:30.706886    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:30.810271 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:30.810238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:31.231290 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.231257    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:31.329947 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.329880    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:31.410766 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.410729    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"]
Sep 16 19:17:31.415537 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.415514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:31.418433 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.418413    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-312c48-dockercfg-2dm77\""
Sep 16 19:17:31.418528 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.418450    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-312c48\""
Sep 16 19:17:31.425203 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.425175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"]
Sep 16 19:17:31.508471 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.508384    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"]
Sep 16 19:17:31.511527 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.511512    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:31.521222 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.521194    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"]
Sep 16 19:17:31.568433 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.568398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6776f0-5de8-4673-a718-25fba7ce06ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x\" (UID: \"bf6776f0-5de8-4673-a718-25fba7ce06ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:31.568602 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.568455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xbxz\" (UniqueName: \"kubernetes.io/projected/bf6776f0-5de8-4673-a718-25fba7ce06ed-kube-api-access-2xbxz\") pod \"payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x\" (UID: \"bf6776f0-5de8-4673-a718-25fba7ce06ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:31.669502 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.669470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6776f0-5de8-4673-a718-25fba7ce06ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x\" (UID: \"bf6776f0-5de8-4673-a718-25fba7ce06ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:31.669704 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.669537    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xbxz\" (UniqueName: \"kubernetes.io/projected/bf6776f0-5de8-4673-a718-25fba7ce06ed-kube-api-access-2xbxz\") pod \"payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x\" (UID: \"bf6776f0-5de8-4673-a718-25fba7ce06ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:31.669704 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.669571    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6mp7\" (UniqueName: \"kubernetes.io/projected/af13074e-7fe4-437d-9f01-f7fccee85b70-kube-api-access-p6mp7\") pod \"payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm\" (UID: \"af13074e-7fe4-437d-9f01-f7fccee85b70\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:31.669704 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.669618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af13074e-7fe4-437d-9f01-f7fccee85b70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm\" (UID: \"af13074e-7fe4-437d-9f01-f7fccee85b70\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:31.670299 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.670281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6776f0-5de8-4673-a718-25fba7ce06ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x\" (UID: \"bf6776f0-5de8-4673-a718-25fba7ce06ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:31.678881 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.678857    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xbxz\" (UniqueName: \"kubernetes.io/projected/bf6776f0-5de8-4673-a718-25fba7ce06ed-kube-api-access-2xbxz\") pod \"payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x\" (UID: \"bf6776f0-5de8-4673-a718-25fba7ce06ed\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:31.726939 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.726914    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:31.770374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.770282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6mp7\" (UniqueName: \"kubernetes.io/projected/af13074e-7fe4-437d-9f01-f7fccee85b70-kube-api-access-p6mp7\") pod \"payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm\" (UID: \"af13074e-7fe4-437d-9f01-f7fccee85b70\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:31.770374 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.770335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af13074e-7fe4-437d-9f01-f7fccee85b70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm\" (UID: \"af13074e-7fe4-437d-9f01-f7fccee85b70\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:31.771086 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.771066    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af13074e-7fe4-437d-9f01-f7fccee85b70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm\" (UID: \"af13074e-7fe4-437d-9f01-f7fccee85b70\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:31.780639 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.779941    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6mp7\" (UniqueName: \"kubernetes.io/projected/af13074e-7fe4-437d-9f01-f7fccee85b70-kube-api-access-p6mp7\") pod \"payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm\" (UID: \"af13074e-7fe4-437d-9f01-f7fccee85b70\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:31.822214 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:31.822179    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:32.005861 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:32.005834    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"]
Sep 16 19:17:32.007385 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:17:32.007362    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6776f0_5de8_4673_a718_25fba7ce06ed.slice/crio-6dd0d37cbd36d378988b36cfd79c564d7f8eef655083c0c47463ef17b003bc6f WatchSource:0}: Error finding container 6dd0d37cbd36d378988b36cfd79c564d7f8eef655083c0c47463ef17b003bc6f: Status 404 returned error can't find the container with id 6dd0d37cbd36d378988b36cfd79c564d7f8eef655083c0c47463ef17b003bc6f
Sep 16 19:17:32.095659 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:32.095630    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"]
Sep 16 19:17:32.097185 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:17:32.097158    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf13074e_7fe4_437d_9f01_f7fccee85b70.slice/crio-0e976403aef35dcd18ff08104a2cbcec18486139c8d2e8d3e901b5e1e79828f7 WatchSource:0}: Error finding container 0e976403aef35dcd18ff08104a2cbcec18486139c8d2e8d3e901b5e1e79828f7: Status 404 returned error can't find the container with id 0e976403aef35dcd18ff08104a2cbcec18486139c8d2e8d3e901b5e1e79828f7
Sep 16 19:17:32.993349 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:32.993310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x" event={"ID":"bf6776f0-5de8-4673-a718-25fba7ce06ed","Type":"ContainerStarted","Data":"16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4"}
Sep 16 19:17:32.993850 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:32.993511    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:32.993850 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:32.993546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x" event={"ID":"bf6776f0-5de8-4673-a718-25fba7ce06ed","Type":"ContainerStarted","Data":"6dd0d37cbd36d378988b36cfd79c564d7f8eef655083c0c47463ef17b003bc6f"}
Sep 16 19:17:32.994835 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:32.994813    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm" event={"ID":"af13074e-7fe4-437d-9f01-f7fccee85b70","Type":"ContainerStarted","Data":"d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768"}
Sep 16 19:17:32.994923 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:32.994855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm" event={"ID":"af13074e-7fe4-437d-9f01-f7fccee85b70","Type":"ContainerStarted","Data":"0e976403aef35dcd18ff08104a2cbcec18486139c8d2e8d3e901b5e1e79828f7"}
Sep 16 19:17:32.994996 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:32.994934    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:33.038719 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:33.038667    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x" podStartSLOduration=2.03865091 podStartE2EDuration="2.03865091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:33.036030176 +0000 UTC m=+2285.404086096" watchObservedRunningTime="2026-09-16 19:17:33.03865091 +0000 UTC m=+2285.406706812"
Sep 16 19:17:37.703269 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:37.703229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:37.800055 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:37.800025    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:44.002087 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:44.002044    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:17:44.002477 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:44.002122    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:17:44.023185 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:44.023130    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm" podStartSLOduration=13.023114212 podStartE2EDuration="13.023114212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:33.070424998 +0000 UTC m=+2285.438480898" watchObservedRunningTime="2026-09-16 19:17:44.023114212 +0000 UTC m=+2296.391170112"
Sep 16 19:17:53.187874 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:53.187841    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:54.019878 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:54.019844    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:55.018258 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:55.018225    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:56.004038 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:56.004004    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:56.580462 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:56.580430    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:56.899039 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:56.898952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:57.768161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:57.768125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:58.699613 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:58.699580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:59.944198 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:17:59.944155    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:00.803547 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:00.803512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:01.615678 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:01.615637    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:02.534275 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:02.534243    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:07.861158 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:07.861126    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:08.511595 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.511563    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:08.554648 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.554615    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"]
Sep 16 19:18:08.554908 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.554878    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm" podUID="af13074e-7fe4-437d-9f01-f7fccee85b70" containerName="payload-processing" containerID="cri-o://d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768" gracePeriod=30
Sep 16 19:18:08.563042 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.562996    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"]
Sep 16 19:18:08.563231 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.563200    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x" podUID="bf6776f0-5de8-4673-a718-25fba7ce06ed" containerName="payload-pre-processing" containerID="cri-o://16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4" gracePeriod=30
Sep 16 19:18:08.606835 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.606806    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:08.680161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.680132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:08.780752 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.780689    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:08.815393 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.815369    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:18:08.837345 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.837326    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:18:08.851355 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.851333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:08.896424 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.896399    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af13074e-7fe4-437d-9f01-f7fccee85b70-plugins-config-volume\") pod \"af13074e-7fe4-437d-9f01-f7fccee85b70\" (UID: \"af13074e-7fe4-437d-9f01-f7fccee85b70\") "
Sep 16 19:18:08.896770 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.896461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6mp7\" (UniqueName: \"kubernetes.io/projected/af13074e-7fe4-437d-9f01-f7fccee85b70-kube-api-access-p6mp7\") pod \"af13074e-7fe4-437d-9f01-f7fccee85b70\" (UID: \"af13074e-7fe4-437d-9f01-f7fccee85b70\") "
Sep 16 19:18:08.896818 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.896765    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af13074e-7fe4-437d-9f01-f7fccee85b70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af13074e-7fe4-437d-9f01-f7fccee85b70" (UID: "af13074e-7fe4-437d-9f01-f7fccee85b70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:08.898436 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.898416    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af13074e-7fe4-437d-9f01-f7fccee85b70-kube-api-access-p6mp7" (OuterVolumeSpecName: "kube-api-access-p6mp7") pod "af13074e-7fe4-437d-9f01-f7fccee85b70" (UID: "af13074e-7fe4-437d-9f01-f7fccee85b70"). InnerVolumeSpecName "kube-api-access-p6mp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:08.996851 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.996819    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xbxz\" (UniqueName: \"kubernetes.io/projected/bf6776f0-5de8-4673-a718-25fba7ce06ed-kube-api-access-2xbxz\") pod \"bf6776f0-5de8-4673-a718-25fba7ce06ed\" (UID: \"bf6776f0-5de8-4673-a718-25fba7ce06ed\") "
Sep 16 19:18:08.997039 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.996867    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6776f0-5de8-4673-a718-25fba7ce06ed-plugins-config-volume\") pod \"bf6776f0-5de8-4673-a718-25fba7ce06ed\" (UID: \"bf6776f0-5de8-4673-a718-25fba7ce06ed\") "
Sep 16 19:18:08.997167 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.997151    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af13074e-7fe4-437d-9f01-f7fccee85b70-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:08.997225 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.997173    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6mp7\" (UniqueName: \"kubernetes.io/projected/af13074e-7fe4-437d-9f01-f7fccee85b70-kube-api-access-p6mp7\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:08.997266 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.997233    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6776f0-5de8-4673-a718-25fba7ce06ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf6776f0-5de8-4673-a718-25fba7ce06ed" (UID: "bf6776f0-5de8-4673-a718-25fba7ce06ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:08.998816 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:08.998801    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6776f0-5de8-4673-a718-25fba7ce06ed-kube-api-access-2xbxz" (OuterVolumeSpecName: "kube-api-access-2xbxz") pod "bf6776f0-5de8-4673-a718-25fba7ce06ed" (UID: "bf6776f0-5de8-4673-a718-25fba7ce06ed"). InnerVolumeSpecName "kube-api-access-2xbxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:09.097756 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.097682    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xbxz\" (UniqueName: \"kubernetes.io/projected/bf6776f0-5de8-4673-a718-25fba7ce06ed-kube-api-access-2xbxz\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:09.097756 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.097711    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6776f0-5de8-4673-a718-25fba7ce06ed-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:09.137579 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.137551    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:09.167225 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.167195    2559 generic.go:356] "Generic (PLEG): container finished" podID="bf6776f0-5de8-4673-a718-25fba7ce06ed" containerID="16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4" exitCode=0
Sep 16 19:18:09.167380 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.167252    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"
Sep 16 19:18:09.167380 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.167277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x" event={"ID":"bf6776f0-5de8-4673-a718-25fba7ce06ed","Type":"ContainerDied","Data":"16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4"}
Sep 16 19:18:09.167380 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.167327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x" event={"ID":"bf6776f0-5de8-4673-a718-25fba7ce06ed","Type":"ContainerDied","Data":"6dd0d37cbd36d378988b36cfd79c564d7f8eef655083c0c47463ef17b003bc6f"}
Sep 16 19:18:09.167380 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.167349    2559 scope.go:117] "RemoveContainer" containerID="16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4"
Sep 16 19:18:09.168669 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.168645    2559 generic.go:356] "Generic (PLEG): container finished" podID="af13074e-7fe4-437d-9f01-f7fccee85b70" containerID="d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768" exitCode=0
Sep 16 19:18:09.168767 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.168695    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"
Sep 16 19:18:09.168874 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.168846    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm" event={"ID":"af13074e-7fe4-437d-9f01-f7fccee85b70","Type":"ContainerDied","Data":"d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768"}
Sep 16 19:18:09.169011 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.168905    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm" event={"ID":"af13074e-7fe4-437d-9f01-f7fccee85b70","Type":"ContainerDied","Data":"0e976403aef35dcd18ff08104a2cbcec18486139c8d2e8d3e901b5e1e79828f7"}
Sep 16 19:18:09.177503 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.177371    2559 scope.go:117] "RemoveContainer" containerID="16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4"
Sep 16 19:18:09.177725 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:18:09.177705    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4\": container with ID starting with 16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4 not found: ID does not exist" containerID="16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4"
Sep 16 19:18:09.177818 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.177739    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4"} err="failed to get container status \"16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4\": rpc error: code = NotFound desc = could not find container \"16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4\": container with ID starting with 16f61a6dc0cbe2fcdb3d362d06241915727a5dbf0267e9dfcf1be9073c278fe4 not found: ID does not exist"
Sep 16 19:18:09.177818 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.177760    2559 scope.go:117] "RemoveContainer" containerID="d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768"
Sep 16 19:18:09.185784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.185766    2559 scope.go:117] "RemoveContainer" containerID="d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768"
Sep 16 19:18:09.186056 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:18:09.186035    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768\": container with ID starting with d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768 not found: ID does not exist" containerID="d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768"
Sep 16 19:18:09.186150 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.186059    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768"} err="failed to get container status \"d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768\": rpc error: code = NotFound desc = could not find container \"d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768\": container with ID starting with d7efe357c778a966b804b997b9d92460ce7f251f4e0e5b94b8260ca2258f4768 not found: ID does not exist"
Sep 16 19:18:09.193605 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.193584    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"]
Sep 16 19:18:09.198995 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.198964    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-312c48-bf779f454-99w9x"]
Sep 16 19:18:09.221379 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.216825    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"]
Sep 16 19:18:09.221379 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.220127    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:09.223364 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:09.223333    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-312c48-5d49d547f7-t6wpm"]
Sep 16 19:18:10.007748 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:10.007716    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:10.222234 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:10.222197    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af13074e-7fe4-437d-9f01-f7fccee85b70" path="/var/lib/kubelet/pods/af13074e-7fe4-437d-9f01-f7fccee85b70/volumes"
Sep 16 19:18:10.222578 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:10.222564    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf6776f0-5de8-4673-a718-25fba7ce06ed" path="/var/lib/kubelet/pods/bf6776f0-5de8-4673-a718-25fba7ce06ed/volumes"
Sep 16 19:18:10.834982 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:10.834951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:11.723783 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:11.723746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:12.613318 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:12.613289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:15.728796 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:15.728747    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:15.815641 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:15.815606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:16.101512 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:16.101431    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:19.578038 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.577999    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"]
Sep 16 19:18:19.578524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.578373    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af13074e-7fe4-437d-9f01-f7fccee85b70" containerName="payload-processing"
Sep 16 19:18:19.578524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.578385    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="af13074e-7fe4-437d-9f01-f7fccee85b70" containerName="payload-processing"
Sep 16 19:18:19.578524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.578405    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6776f0-5de8-4673-a718-25fba7ce06ed" containerName="payload-pre-processing"
Sep 16 19:18:19.578524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.578411    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6776f0-5de8-4673-a718-25fba7ce06ed" containerName="payload-pre-processing"
Sep 16 19:18:19.578524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.578461    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6776f0-5de8-4673-a718-25fba7ce06ed" containerName="payload-pre-processing"
Sep 16 19:18:19.578524 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.578467    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="af13074e-7fe4-437d-9f01-f7fccee85b70" containerName="payload-processing"
Sep 16 19:18:19.582831 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.582812    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"
Sep 16 19:18:19.586196 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.586174    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:18:19.586313 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.586198    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:18:19.590991 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.590954    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"]
Sep 16 19:18:19.685086 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.685052    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c9b215-3647-427e-824f-b48a244bf9e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc\" (UID: \"b5c9b215-3647-427e-824f-b48a244bf9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"
Sep 16 19:18:19.785703 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.785668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c9b215-3647-427e-824f-b48a244bf9e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc\" (UID: \"b5c9b215-3647-427e-824f-b48a244bf9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"
Sep 16 19:18:19.786291 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.786273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c9b215-3647-427e-824f-b48a244bf9e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc\" (UID: \"b5c9b215-3647-427e-824f-b48a244bf9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"
Sep 16 19:18:19.879434 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.879330    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:19.894388 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:19.894353    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"
Sep 16 19:18:20.167678 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:20.167652    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"]
Sep 16 19:18:20.169993 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:18:20.169953    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c9b215_3647_427e_824f_b48a244bf9e9.slice/crio-6e5a65e349f4f1216826f6908fdf1f4a11607e4ee47caecc98f840c622e516b4 WatchSource:0}: Error finding container 6e5a65e349f4f1216826f6908fdf1f4a11607e4ee47caecc98f840c622e516b4: Status 404 returned error can't find the container with id 6e5a65e349f4f1216826f6908fdf1f4a11607e4ee47caecc98f840c622e516b4
Sep 16 19:18:20.223315 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:20.223277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc" event={"ID":"b5c9b215-3647-427e-824f-b48a244bf9e9","Type":"ContainerStarted","Data":"6e5a65e349f4f1216826f6908fdf1f4a11607e4ee47caecc98f840c622e516b4"}
Sep 16 19:18:20.804428 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:20.804374    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:21.229421 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:21.229385    2559 generic.go:356] "Generic (PLEG): container finished" podID="b5c9b215-3647-427e-824f-b48a244bf9e9" containerID="a62332ea90c9801f91139790fd605b3724b899efd03e91599df278d348be4265" exitCode=0
Sep 16 19:18:21.229587 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:21.229470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc" event={"ID":"b5c9b215-3647-427e-824f-b48a244bf9e9","Type":"ContainerDied","Data":"a62332ea90c9801f91139790fd605b3724b899efd03e91599df278d348be4265"}
Sep 16 19:18:21.734902 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:21.734867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:22.373621 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:22.373597    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"
Sep 16 19:18:22.512177 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:22.512090    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c9b215-3647-427e-824f-b48a244bf9e9-maas-api-service-ca\") pod \"b5c9b215-3647-427e-824f-b48a244bf9e9\" (UID: \"b5c9b215-3647-427e-824f-b48a244bf9e9\") "
Sep 16 19:18:22.512452 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:22.512426    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c9b215-3647-427e-824f-b48a244bf9e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b5c9b215-3647-427e-824f-b48a244bf9e9" (UID: "b5c9b215-3647-427e-824f-b48a244bf9e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:22.613371 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:22.613338    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c9b215-3647-427e-824f-b48a244bf9e9-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:22.730669 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:22.730636    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:23.241638 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:23.241597    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc" event={"ID":"b5c9b215-3647-427e-824f-b48a244bf9e9","Type":"ContainerDied","Data":"6e5a65e349f4f1216826f6908fdf1f4a11607e4ee47caecc98f840c622e516b4"}
Sep 16 19:18:23.241638 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:23.241629    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"
Sep 16 19:18:23.241850 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:23.241667    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e5a65e349f4f1216826f6908fdf1f4a11607e4ee47caecc98f840c622e516b4"
Sep 16 19:18:23.900347 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:23.900313    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:24.877897 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:24.877868    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:28.426768 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:28.426738    2559 scope.go:117] "RemoveContainer" containerID="ecd2a79e3194f10d22000a8b6ff6aef712a6311c183595f14365d55554a73c51"
Sep 16 19:18:28.434800 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:28.434777    2559 scope.go:117] "RemoveContainer" containerID="b0587f84cd884ce5b61ed36aa7fd5972ebc243f0a713dbed054059814aa01db8"
Sep 16 19:18:35.245533 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.245503    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:35.296833 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.296801    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"]
Sep 16 19:18:35.297051 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.297014    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f" podUID="07555e40-8c8f-48ea-9b6e-5bf102970a40" containerName="payload-processing" containerID="cri-o://9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80" gracePeriod=30
Sep 16 19:18:35.320520 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.320489    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"]
Sep 16 19:18:35.320696 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.320673    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f" podUID="256a2a1a-3820-4e16-b631-ad5f1517036a" containerName="payload-pre-processing" containerID="cri-o://d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a" gracePeriod=30
Sep 16 19:18:35.350037 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.350000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:35.449640 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.449606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:35.556707 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.556678    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:18:35.585140 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.585112    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:18:35.593893 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.593865    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:35.616915 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.616887    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a2a1a-3820-4e16-b631-ad5f1517036a-plugins-config-volume\") pod \"256a2a1a-3820-4e16-b631-ad5f1517036a\" (UID: \"256a2a1a-3820-4e16-b631-ad5f1517036a\") "
Sep 16 19:18:35.617124 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.616940    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07555e40-8c8f-48ea-9b6e-5bf102970a40-plugins-config-volume\") pod \"07555e40-8c8f-48ea-9b6e-5bf102970a40\" (UID: \"07555e40-8c8f-48ea-9b6e-5bf102970a40\") "
Sep 16 19:18:35.617124 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.616964    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4v82\" (UniqueName: \"kubernetes.io/projected/256a2a1a-3820-4e16-b631-ad5f1517036a-kube-api-access-l4v82\") pod \"256a2a1a-3820-4e16-b631-ad5f1517036a\" (UID: \"256a2a1a-3820-4e16-b631-ad5f1517036a\") "
Sep 16 19:18:35.617124 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.617033    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pntrb\" (UniqueName: \"kubernetes.io/projected/07555e40-8c8f-48ea-9b6e-5bf102970a40-kube-api-access-pntrb\") pod \"07555e40-8c8f-48ea-9b6e-5bf102970a40\" (UID: \"07555e40-8c8f-48ea-9b6e-5bf102970a40\") "
Sep 16 19:18:35.617348 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.617324    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07555e40-8c8f-48ea-9b6e-5bf102970a40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07555e40-8c8f-48ea-9b6e-5bf102970a40" (UID: "07555e40-8c8f-48ea-9b6e-5bf102970a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:35.617348 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.617331    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256a2a1a-3820-4e16-b631-ad5f1517036a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "256a2a1a-3820-4e16-b631-ad5f1517036a" (UID: "256a2a1a-3820-4e16-b631-ad5f1517036a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:35.619116 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.619090    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07555e40-8c8f-48ea-9b6e-5bf102970a40-kube-api-access-pntrb" (OuterVolumeSpecName: "kube-api-access-pntrb") pod "07555e40-8c8f-48ea-9b6e-5bf102970a40" (UID: "07555e40-8c8f-48ea-9b6e-5bf102970a40"). InnerVolumeSpecName "kube-api-access-pntrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:35.619214 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.619177    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256a2a1a-3820-4e16-b631-ad5f1517036a-kube-api-access-l4v82" (OuterVolumeSpecName: "kube-api-access-l4v82") pod "256a2a1a-3820-4e16-b631-ad5f1517036a" (UID: "256a2a1a-3820-4e16-b631-ad5f1517036a"). InnerVolumeSpecName "kube-api-access-l4v82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:35.647241 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.647219    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"]
Sep 16 19:18:35.653151 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.653122    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4992ef-bpmtc"]
Sep 16 19:18:35.681525 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.681500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:35.717650 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.717623    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a2a1a-3820-4e16-b631-ad5f1517036a-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:35.717730 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.717650    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07555e40-8c8f-48ea-9b6e-5bf102970a40-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:35.717730 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.717661    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4v82\" (UniqueName: \"kubernetes.io/projected/256a2a1a-3820-4e16-b631-ad5f1517036a-kube-api-access-l4v82\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:35.717730 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.717671    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pntrb\" (UniqueName: \"kubernetes.io/projected/07555e40-8c8f-48ea-9b6e-5bf102970a40-kube-api-access-pntrb\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:35.981671 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:35.981640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:36.222335 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.222297    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5c9b215-3647-427e-824f-b48a244bf9e9" path="/var/lib/kubelet/pods/b5c9b215-3647-427e-824f-b48a244bf9e9/volumes"
Sep 16 19:18:36.258689 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.258661    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:36.309622 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.309587    2559 generic.go:356] "Generic (PLEG): container finished" podID="256a2a1a-3820-4e16-b631-ad5f1517036a" containerID="d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a" exitCode=0
Sep 16 19:18:36.309789 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.309640    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"
Sep 16 19:18:36.309789 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.309669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f" event={"ID":"256a2a1a-3820-4e16-b631-ad5f1517036a","Type":"ContainerDied","Data":"d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a"}
Sep 16 19:18:36.309789 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.309736    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f" event={"ID":"256a2a1a-3820-4e16-b631-ad5f1517036a","Type":"ContainerDied","Data":"3d4d9fe1a5755b11db36d4ad7ed48b9dccb165c0df017ac7640dfe629b8ab7bd"}
Sep 16 19:18:36.309789 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.309771    2559 scope.go:117] "RemoveContainer" containerID="d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a"
Sep 16 19:18:36.311284 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.311243    2559 generic.go:356] "Generic (PLEG): container finished" podID="07555e40-8c8f-48ea-9b6e-5bf102970a40" containerID="9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80" exitCode=0
Sep 16 19:18:36.311361 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.311293    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"
Sep 16 19:18:36.311361 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.311313    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f" event={"ID":"07555e40-8c8f-48ea-9b6e-5bf102970a40","Type":"ContainerDied","Data":"9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80"}
Sep 16 19:18:36.311361 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.311355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f" event={"ID":"07555e40-8c8f-48ea-9b6e-5bf102970a40","Type":"ContainerDied","Data":"dfd26d48b88911e76385e914c2a5ddb52951eeaa27cce52c4ff722f592405d75"}
Sep 16 19:18:36.320896 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.320876    2559 scope.go:117] "RemoveContainer" containerID="d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a"
Sep 16 19:18:36.321191 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:18:36.321171    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a\": container with ID starting with d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a not found: ID does not exist" containerID="d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a"
Sep 16 19:18:36.321262 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.321198    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a"} err="failed to get container status \"d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a\": rpc error: code = NotFound desc = could not find container \"d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a\": container with ID starting with d02d11d0d01d10737e3a8ba92ddf017f963a3387bd7043d2de6d3c089c5fc04a not found: ID does not exist"
Sep 16 19:18:36.321262 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.321218    2559 scope.go:117] "RemoveContainer" containerID="9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80"
Sep 16 19:18:36.328794 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.328772    2559 scope.go:117] "RemoveContainer" containerID="9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80"
Sep 16 19:18:36.329060 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:18:36.329042    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80\": container with ID starting with 9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80 not found: ID does not exist" containerID="9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80"
Sep 16 19:18:36.329114 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.329067    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80"} err="failed to get container status \"9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80\": rpc error: code = NotFound desc = could not find container \"9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80\": container with ID starting with 9057c7ae93128a1b7faebcab7d1b29a96b12333cbaae7876f9f3f724033cec80 not found: ID does not exist"
Sep 16 19:18:36.337072 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.337051    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"]
Sep 16 19:18:36.346649 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.346628    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4992ef-64f8ddbcd-nfj8f"]
Sep 16 19:18:36.358099 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.358077    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"]
Sep 16 19:18:36.362488 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:36.362466    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4992ef-7d674df7d7-sg26f"]
Sep 16 19:18:37.067503 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:37.067469    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:38.156028 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:38.155994    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:38.226825 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:38.226792    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07555e40-8c8f-48ea-9b6e-5bf102970a40" path="/var/lib/kubelet/pods/07555e40-8c8f-48ea-9b6e-5bf102970a40/volumes"
Sep 16 19:18:38.227167 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:38.227153    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="256a2a1a-3820-4e16-b631-ad5f1517036a" path="/var/lib/kubelet/pods/256a2a1a-3820-4e16-b631-ad5f1517036a/volumes"
Sep 16 19:18:39.331519 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:39.331489    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:39.702554 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:39.702479    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:39.803676 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:39.803643    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:40.969818 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:40.969783    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"]
Sep 16 19:18:41.066637 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:41.066606    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:55.988452 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:55.988415    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp" podUID="7ce62def-233a-4304-b6c9-09caf79d59d5" containerName="main" containerID="cri-o://5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729" gracePeriod=45
Sep 16 19:18:56.218333 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.218301    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:18:56.263820 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.263740    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-home\") pod \"7ce62def-233a-4304-b6c9-09caf79d59d5\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") "
Sep 16 19:18:56.263820 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.263806    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2jzs\" (UniqueName: \"kubernetes.io/projected/7ce62def-233a-4304-b6c9-09caf79d59d5-kube-api-access-f2jzs\") pod \"7ce62def-233a-4304-b6c9-09caf79d59d5\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") "
Sep 16 19:18:56.264062 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.263847    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-tmp-dir\") pod \"7ce62def-233a-4304-b6c9-09caf79d59d5\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") "
Sep 16 19:18:56.264062 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.263902    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-model-cache\") pod \"7ce62def-233a-4304-b6c9-09caf79d59d5\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") "
Sep 16 19:18:56.264062 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.263927    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce62def-233a-4304-b6c9-09caf79d59d5-tls-certs\") pod \"7ce62def-233a-4304-b6c9-09caf79d59d5\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") "
Sep 16 19:18:56.264062 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.263961    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-dshm\") pod \"7ce62def-233a-4304-b6c9-09caf79d59d5\" (UID: \"7ce62def-233a-4304-b6c9-09caf79d59d5\") "
Sep 16 19:18:56.264062 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.264018    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-home" (OuterVolumeSpecName: "home") pod "7ce62def-233a-4304-b6c9-09caf79d59d5" (UID: "7ce62def-233a-4304-b6c9-09caf79d59d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.264376 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.264172    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ce62def-233a-4304-b6c9-09caf79d59d5" (UID: "7ce62def-233a-4304-b6c9-09caf79d59d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.264376 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.264197    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ce62def-233a-4304-b6c9-09caf79d59d5" (UID: "7ce62def-233a-4304-b6c9-09caf79d59d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.264376 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.264270    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.264376 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.264287    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.264376 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.264300    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.266381 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.266354    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ce62def-233a-4304-b6c9-09caf79d59d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ce62def-233a-4304-b6c9-09caf79d59d5" (UID: "7ce62def-233a-4304-b6c9-09caf79d59d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:56.266490 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.266478    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-dshm" (OuterVolumeSpecName: "dshm") pod "7ce62def-233a-4304-b6c9-09caf79d59d5" (UID: "7ce62def-233a-4304-b6c9-09caf79d59d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.266546 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.266498    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce62def-233a-4304-b6c9-09caf79d59d5-kube-api-access-f2jzs" (OuterVolumeSpecName: "kube-api-access-f2jzs") pod "7ce62def-233a-4304-b6c9-09caf79d59d5" (UID: "7ce62def-233a-4304-b6c9-09caf79d59d5"). InnerVolumeSpecName "kube-api-access-f2jzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:56.365489 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.365454    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ce62def-233a-4304-b6c9-09caf79d59d5-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.365489 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.365481    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ce62def-233a-4304-b6c9-09caf79d59d5-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.365489 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.365490    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2jzs\" (UniqueName: \"kubernetes.io/projected/7ce62def-233a-4304-b6c9-09caf79d59d5-kube-api-access-f2jzs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.406226 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.406191    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ce62def-233a-4304-b6c9-09caf79d59d5" containerID="5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729" exitCode=0
Sep 16 19:18:56.406369 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.406236    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"
Sep 16 19:18:56.406369 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.406246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp" event={"ID":"7ce62def-233a-4304-b6c9-09caf79d59d5","Type":"ContainerDied","Data":"5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729"}
Sep 16 19:18:56.406369 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.406285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp" event={"ID":"7ce62def-233a-4304-b6c9-09caf79d59d5","Type":"ContainerDied","Data":"d0a327b4a2a2566b0b29237e98e2c690cb64a233b7b7c97bf3defdda09a368eb"}
Sep 16 19:18:56.406369 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.406307    2559 scope.go:117] "RemoveContainer" containerID="5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729"
Sep 16 19:18:56.415465 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.415446    2559 scope.go:117] "RemoveContainer" containerID="5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729"
Sep 16 19:18:56.415724 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:18:56.415706    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729\": container with ID starting with 5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729 not found: ID does not exist" containerID="5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729"
Sep 16 19:18:56.415791 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.415731    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729"} err="failed to get container status \"5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729\": rpc error: code = NotFound desc = could not find container \"5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729\": container with ID starting with 5b4daed2d7afc80b1bebcff638fce18f3eb4aa3a2b22fa6bb859653debdd3729 not found: ID does not exist"
Sep 16 19:18:56.434723 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.434689    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"]
Sep 16 19:18:56.440346 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:56.440321    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-4992ef/ipp-route-4992ef-kserve-5d4896675-t78dp"]
Sep 16 19:18:58.221519 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:18:58.221481    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ce62def-233a-4304-b6c9-09caf79d59d5" path="/var/lib/kubelet/pods/7ce62def-233a-4304-b6c9-09caf79d59d5/volumes"
Sep 16 19:19:07.432199 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:07.432145    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:17.508914 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:17.508879    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:19.506455 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:19.506415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:25.185054 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:25.185016    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:25.948466 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:25.948415    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:28.152068 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:28.152037    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:19:28.158849 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:28.158827    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:19:28.461980 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:28.461950    2559 scope.go:117] "RemoveContainer" containerID="98711ff655a30b2962d88e71764796ee7c703a211b5fe59697c870c7cfb867ad"
Sep 16 19:19:32.499261 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:32.499228    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:32.619706 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:32.619669    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:32.905989 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:32.905889    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:33.295660 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.295631    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"]
Sep 16 19:19:33.295938 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.295927    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="256a2a1a-3820-4e16-b631-ad5f1517036a" containerName="payload-pre-processing"
Sep 16 19:19:33.295996 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.295940    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="256a2a1a-3820-4e16-b631-ad5f1517036a" containerName="payload-pre-processing"
Sep 16 19:19:33.295996 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.295952    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ce62def-233a-4304-b6c9-09caf79d59d5" containerName="main"
Sep 16 19:19:33.295996 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.295957    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ce62def-233a-4304-b6c9-09caf79d59d5" containerName="main"
Sep 16 19:19:33.295996 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.295989    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07555e40-8c8f-48ea-9b6e-5bf102970a40" containerName="payload-processing"
Sep 16 19:19:33.295996 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.295994    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="07555e40-8c8f-48ea-9b6e-5bf102970a40" containerName="payload-processing"
Sep 16 19:19:33.296161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.296002    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c9b215-3647-427e-824f-b48a244bf9e9" containerName="revoke-keys"
Sep 16 19:19:33.296161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.296008    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c9b215-3647-427e-824f-b48a244bf9e9" containerName="revoke-keys"
Sep 16 19:19:33.296161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.296055    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ce62def-233a-4304-b6c9-09caf79d59d5" containerName="main"
Sep 16 19:19:33.296161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.296065    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="07555e40-8c8f-48ea-9b6e-5bf102970a40" containerName="payload-processing"
Sep 16 19:19:33.296161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.296071    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="256a2a1a-3820-4e16-b631-ad5f1517036a" containerName="payload-pre-processing"
Sep 16 19:19:33.296161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.296078    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c9b215-3647-427e-824f-b48a244bf9e9" containerName="revoke-keys"
Sep 16 19:19:33.300644 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.300624    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"
Sep 16 19:19:33.307783 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.307763    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:19:33.308161 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.308144    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:19:33.316432 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.316410    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"]
Sep 16 19:19:33.474297 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.474253    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5604b59a-09f6-4ae5-81e4-69160588c020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q\" (UID: \"5604b59a-09f6-4ae5-81e4-69160588c020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"
Sep 16 19:19:33.575106 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.575011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5604b59a-09f6-4ae5-81e4-69160588c020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q\" (UID: \"5604b59a-09f6-4ae5-81e4-69160588c020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"
Sep 16 19:19:33.575662 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.575641    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5604b59a-09f6-4ae5-81e4-69160588c020-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q\" (UID: \"5604b59a-09f6-4ae5-81e4-69160588c020\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"
Sep 16 19:19:33.588390 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.588344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:33.610300 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.610275    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"
Sep 16 19:19:33.880864 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:33.880702    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"]
Sep 16 19:19:33.883254 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:19:33.883229    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5604b59a_09f6_4ae5_81e4_69160588c020.slice/crio-942867e7be2e9d3d79291a9848f109facada963bac92faf87dae525f70274af5 WatchSource:0}: Error finding container 942867e7be2e9d3d79291a9848f109facada963bac92faf87dae525f70274af5: Status 404 returned error can't find the container with id 942867e7be2e9d3d79291a9848f109facada963bac92faf87dae525f70274af5
Sep 16 19:19:34.365441 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:34.365411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:34.577212 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:34.577186    2559 generic.go:356] "Generic (PLEG): container finished" podID="5604b59a-09f6-4ae5-81e4-69160588c020" containerID="fe1ddec1e20e4f8e500372e4880ca633478068a765909f087432a1da5418e44b" exitCode=0
Sep 16 19:19:34.577512 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:34.577252    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q" event={"ID":"5604b59a-09f6-4ae5-81e4-69160588c020","Type":"ContainerDied","Data":"fe1ddec1e20e4f8e500372e4880ca633478068a765909f087432a1da5418e44b"}
Sep 16 19:19:34.577512 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:34.577306    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q" event={"ID":"5604b59a-09f6-4ae5-81e4-69160588c020","Type":"ContainerStarted","Data":"942867e7be2e9d3d79291a9848f109facada963bac92faf87dae525f70274af5"}
Sep 16 19:19:35.349454 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:35.349398    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:35.706021 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:35.705991    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"
Sep 16 19:19:35.791958 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:35.791925    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5604b59a-09f6-4ae5-81e4-69160588c020-maas-api-service-ca\") pod \"5604b59a-09f6-4ae5-81e4-69160588c020\" (UID: \"5604b59a-09f6-4ae5-81e4-69160588c020\") "
Sep 16 19:19:35.792325 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:35.792303    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5604b59a-09f6-4ae5-81e4-69160588c020-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5604b59a-09f6-4ae5-81e4-69160588c020" (UID: "5604b59a-09f6-4ae5-81e4-69160588c020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:35.893373 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:35.893270    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5604b59a-09f6-4ae5-81e4-69160588c020-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:19:36.228278 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:36.228249    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:36.587264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:36.587179    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"
Sep 16 19:19:36.587264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:36.587188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q" event={"ID":"5604b59a-09f6-4ae5-81e4-69160588c020","Type":"ContainerDied","Data":"942867e7be2e9d3d79291a9848f109facada963bac92faf87dae525f70274af5"}
Sep 16 19:19:36.587264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:36.587234    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="942867e7be2e9d3d79291a9848f109facada963bac92faf87dae525f70274af5"
Sep 16 19:19:37.258149 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:37.258114    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:43.826145 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:43.826111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:43.863624 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:43.863594    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"]
Sep 16 19:19:43.863863 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:43.863834    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz" podUID="c61de80a-ace8-4f43-98ad-bd4eaf78c818" containerName="payload-processing" containerID="cri-o://3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6" gracePeriod=30
Sep 16 19:19:43.872571 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:43.872352    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"]
Sep 16 19:19:43.872691 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:43.872572    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h" podUID="867e5c3a-d02a-4cea-9d22-e76c2973da40" containerName="payload-pre-processing" containerID="cri-o://30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2" gracePeriod=30
Sep 16 19:19:43.916852 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:43.916817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:43.983606 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:43.983580    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:44.096369 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.096333    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:44.110984 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.110953    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:19:44.118415 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.118389    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"]
Sep 16 19:19:44.129437 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.129414    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-766919-n6p8q"]
Sep 16 19:19:44.136412 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.136394    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:19:44.163905 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.163883    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gc7j\" (UniqueName: \"kubernetes.io/projected/867e5c3a-d02a-4cea-9d22-e76c2973da40-kube-api-access-8gc7j\") pod \"867e5c3a-d02a-4cea-9d22-e76c2973da40\" (UID: \"867e5c3a-d02a-4cea-9d22-e76c2973da40\") "
Sep 16 19:19:44.164062 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.163920    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61de80a-ace8-4f43-98ad-bd4eaf78c818-plugins-config-volume\") pod \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\" (UID: \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\") "
Sep 16 19:19:44.164062 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.164004    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867e5c3a-d02a-4cea-9d22-e76c2973da40-plugins-config-volume\") pod \"867e5c3a-d02a-4cea-9d22-e76c2973da40\" (UID: \"867e5c3a-d02a-4cea-9d22-e76c2973da40\") "
Sep 16 19:19:44.164181 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.164066    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhz6p\" (UniqueName: \"kubernetes.io/projected/c61de80a-ace8-4f43-98ad-bd4eaf78c818-kube-api-access-nhz6p\") pod \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\" (UID: \"c61de80a-ace8-4f43-98ad-bd4eaf78c818\") "
Sep 16 19:19:44.164344 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.164321    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61de80a-ace8-4f43-98ad-bd4eaf78c818-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c61de80a-ace8-4f43-98ad-bd4eaf78c818" (UID: "c61de80a-ace8-4f43-98ad-bd4eaf78c818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:44.164414 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.164363    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867e5c3a-d02a-4cea-9d22-e76c2973da40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "867e5c3a-d02a-4cea-9d22-e76c2973da40" (UID: "867e5c3a-d02a-4cea-9d22-e76c2973da40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:44.165983 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.165946    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867e5c3a-d02a-4cea-9d22-e76c2973da40-kube-api-access-8gc7j" (OuterVolumeSpecName: "kube-api-access-8gc7j") pod "867e5c3a-d02a-4cea-9d22-e76c2973da40" (UID: "867e5c3a-d02a-4cea-9d22-e76c2973da40"). InnerVolumeSpecName "kube-api-access-8gc7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:44.166093 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.166051    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61de80a-ace8-4f43-98ad-bd4eaf78c818-kube-api-access-nhz6p" (OuterVolumeSpecName: "kube-api-access-nhz6p") pod "c61de80a-ace8-4f43-98ad-bd4eaf78c818" (UID: "c61de80a-ace8-4f43-98ad-bd4eaf78c818"). InnerVolumeSpecName "kube-api-access-nhz6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:44.201543 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.201517    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:44.221918 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.221891    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5604b59a-09f6-4ae5-81e4-69160588c020" path="/var/lib/kubelet/pods/5604b59a-09f6-4ae5-81e4-69160588c020/volumes"
Sep 16 19:19:44.265203 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.265176    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867e5c3a-d02a-4cea-9d22-e76c2973da40-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:19:44.265203 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.265200    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhz6p\" (UniqueName: \"kubernetes.io/projected/c61de80a-ace8-4f43-98ad-bd4eaf78c818-kube-api-access-nhz6p\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:19:44.265331 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.265217    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gc7j\" (UniqueName: \"kubernetes.io/projected/867e5c3a-d02a-4cea-9d22-e76c2973da40-kube-api-access-8gc7j\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:19:44.265331 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.265229    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61de80a-ace8-4f43-98ad-bd4eaf78c818-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:19:44.530130 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.530100    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:44.624661 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.624625    2559 generic.go:356] "Generic (PLEG): container finished" podID="867e5c3a-d02a-4cea-9d22-e76c2973da40" containerID="30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2" exitCode=0
Sep 16 19:19:44.624851 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.624676    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"
Sep 16 19:19:44.624851 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.624705    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h" event={"ID":"867e5c3a-d02a-4cea-9d22-e76c2973da40","Type":"ContainerDied","Data":"30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2"}
Sep 16 19:19:44.624851 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.624763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h" event={"ID":"867e5c3a-d02a-4cea-9d22-e76c2973da40","Type":"ContainerDied","Data":"23e7d91a713e7c79b4924678ce023c553b346010547993341678ff117953f507"}
Sep 16 19:19:44.624851 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.624799    2559 scope.go:117] "RemoveContainer" containerID="30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2"
Sep 16 19:19:44.626199 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.626175    2559 generic.go:356] "Generic (PLEG): container finished" podID="c61de80a-ace8-4f43-98ad-bd4eaf78c818" containerID="3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6" exitCode=0
Sep 16 19:19:44.626307 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.626231    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"
Sep 16 19:19:44.626307 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.626255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz" event={"ID":"c61de80a-ace8-4f43-98ad-bd4eaf78c818","Type":"ContainerDied","Data":"3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6"}
Sep 16 19:19:44.626398 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.626320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz" event={"ID":"c61de80a-ace8-4f43-98ad-bd4eaf78c818","Type":"ContainerDied","Data":"7a43e62e6df3ec93b24f2bd2a898edad2b5c7f66ea7b4a3f8c7e6d1a56c73fa3"}
Sep 16 19:19:44.636030 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.635960    2559 scope.go:117] "RemoveContainer" containerID="30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2"
Sep 16 19:19:44.636495 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:19:44.636466    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2\": container with ID starting with 30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2 not found: ID does not exist" containerID="30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2"
Sep 16 19:19:44.636585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.636493    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2"} err="failed to get container status \"30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2\": rpc error: code = NotFound desc = could not find container \"30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2\": container with ID starting with 30e5519246966bcba2921b178ad9072549206712d70ba7a4d486e2e0ae0b44c2 not found: ID does not exist"
Sep 16 19:19:44.636585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.636508    2559 scope.go:117] "RemoveContainer" containerID="3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6"
Sep 16 19:19:44.644551 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.644504    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"]
Sep 16 19:19:44.644611 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.644570    2559 scope.go:117] "RemoveContainer" containerID="3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6"
Sep 16 19:19:44.644865 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:19:44.644832    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6\": container with ID starting with 3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6 not found: ID does not exist" containerID="3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6"
Sep 16 19:19:44.644928 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.644870    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6"} err="failed to get container status \"3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6\": rpc error: code = NotFound desc = could not find container \"3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6\": container with ID starting with 3a6821e822480a8a5a7013aeaf38b988e389108b8512254c5eba8b4a1b3875c6 not found: ID does not exist"
Sep 16 19:19:44.649994 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.649960    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-766919-6f547b9dc8-m692h"]
Sep 16 19:19:44.651359 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.651340    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:44.661999 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.661960    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"]
Sep 16 19:19:44.668345 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:44.668325    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-766919-6f8b985f75-h9dzz"]
Sep 16 19:19:45.701310 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:45.701265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:46.222151 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:46.222119    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="867e5c3a-d02a-4cea-9d22-e76c2973da40" path="/var/lib/kubelet/pods/867e5c3a-d02a-4cea-9d22-e76c2973da40/volumes"
Sep 16 19:19:46.222496 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:46.222482    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c61de80a-ace8-4f43-98ad-bd4eaf78c818" path="/var/lib/kubelet/pods/c61de80a-ace8-4f43-98ad-bd4eaf78c818/volumes"
Sep 16 19:19:46.487998 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:46.487882    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:50.926286 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:50.926203    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:51.036710 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.036678    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:51.275002 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.274957    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:51.583527 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583443    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"]
Sep 16 19:19:51.583741 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583730    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c61de80a-ace8-4f43-98ad-bd4eaf78c818" containerName="payload-processing"
Sep 16 19:19:51.583795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583743    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c61de80a-ace8-4f43-98ad-bd4eaf78c818" containerName="payload-processing"
Sep 16 19:19:51.583795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583764    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867e5c3a-d02a-4cea-9d22-e76c2973da40" containerName="payload-pre-processing"
Sep 16 19:19:51.583795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583770    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="867e5c3a-d02a-4cea-9d22-e76c2973da40" containerName="payload-pre-processing"
Sep 16 19:19:51.583795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583782    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5604b59a-09f6-4ae5-81e4-69160588c020" containerName="revoke-keys"
Sep 16 19:19:51.583795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583787    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5604b59a-09f6-4ae5-81e4-69160588c020" containerName="revoke-keys"
Sep 16 19:19:51.583940 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583833    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c61de80a-ace8-4f43-98ad-bd4eaf78c818" containerName="payload-processing"
Sep 16 19:19:51.583940 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583840    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="867e5c3a-d02a-4cea-9d22-e76c2973da40" containerName="payload-pre-processing"
Sep 16 19:19:51.583940 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.583849    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5604b59a-09f6-4ae5-81e4-69160588c020" containerName="revoke-keys"
Sep 16 19:19:51.588222 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.588205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"
Sep 16 19:19:51.591254 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.591232    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:19:51.591390 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.591279    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:19:51.599858 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.599835    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"]
Sep 16 19:19:51.626412 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.626388    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee68d-c45e-4f4b-9dc5-7b234dfb081d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm\" (UID: \"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"
Sep 16 19:19:51.727691 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.727654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee68d-c45e-4f4b-9dc5-7b234dfb081d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm\" (UID: \"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"
Sep 16 19:19:51.728334 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.728310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee68d-c45e-4f4b-9dc5-7b234dfb081d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm\" (UID: \"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"
Sep 16 19:19:51.898580 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:51.898496    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"
Sep 16 19:19:52.170590 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:52.170564    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"]
Sep 16 19:19:52.173357 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:19:52.173321    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dee68d_c45e_4f4b_9dc5_7b234dfb081d.slice/crio-7b0945b5f505be8cd3fd8566c9a863e4339a67c7dd251be8e692831e4d22e321 WatchSource:0}: Error finding container 7b0945b5f505be8cd3fd8566c9a863e4339a67c7dd251be8e692831e4d22e321: Status 404 returned error can't find the container with id 7b0945b5f505be8cd3fd8566c9a863e4339a67c7dd251be8e692831e4d22e321
Sep 16 19:19:52.472378 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:52.472343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:52.664143 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:52.664111    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm" event={"ID":"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d","Type":"ContainerStarted","Data":"bd7270cd24de01cf0bedfcd76d64baebaf6b21a88f21ff1ebcb89052bd83cfb2"}
Sep 16 19:19:52.664306 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:52.664166    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm" event={"ID":"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d","Type":"ContainerStarted","Data":"7b0945b5f505be8cd3fd8566c9a863e4339a67c7dd251be8e692831e4d22e321"}
Sep 16 19:19:52.682906 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:52.682849    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm" podStartSLOduration=1.682832131 podStartE2EDuration="1.682832131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:52.681396446 +0000 UTC m=+2425.049452348" watchObservedRunningTime="2026-09-16 19:19:52.682832131 +0000 UTC m=+2425.050888031"
Sep 16 19:19:53.395088 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:53.395051    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:53.670295 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:53.670205    2559 generic.go:356] "Generic (PLEG): container finished" podID="d6dee68d-c45e-4f4b-9dc5-7b234dfb081d" containerID="bd7270cd24de01cf0bedfcd76d64baebaf6b21a88f21ff1ebcb89052bd83cfb2" exitCode=0
Sep 16 19:19:53.670446 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:53.670296    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm" event={"ID":"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d","Type":"ContainerDied","Data":"bd7270cd24de01cf0bedfcd76d64baebaf6b21a88f21ff1ebcb89052bd83cfb2"}
Sep 16 19:19:54.319214 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:54.319179    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:54.797745 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:54.797722    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"
Sep 16 19:19:54.853244 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:54.853218    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee68d-c45e-4f4b-9dc5-7b234dfb081d-maas-api-service-ca\") pod \"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d\" (UID: \"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d\") "
Sep 16 19:19:54.853548 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:54.853523    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6dee68d-c45e-4f4b-9dc5-7b234dfb081d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6dee68d-c45e-4f4b-9dc5-7b234dfb081d" (UID: "d6dee68d-c45e-4f4b-9dc5-7b234dfb081d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:54.954706 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:54.954630    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6dee68d-c45e-4f4b-9dc5-7b234dfb081d-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:19:55.680250 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:55.680219    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"
Sep 16 19:19:55.680430 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:55.680214    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm" event={"ID":"d6dee68d-c45e-4f4b-9dc5-7b234dfb081d","Type":"ContainerDied","Data":"7b0945b5f505be8cd3fd8566c9a863e4339a67c7dd251be8e692831e4d22e321"}
Sep 16 19:19:55.680430 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:19:55.680339    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0945b5f505be8cd3fd8566c9a863e4339a67c7dd251be8e692831e4d22e321"
Sep 16 19:20:02.202807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.202772    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"]
Sep 16 19:20:02.203449 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.203246    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:02.205532 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.205509    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ca3f9a-29826435-jm6p9"]
Sep 16 19:20:02.222617 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.222589    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcb8c646-53fc-462c-8146-476db2e5118d" path="/var/lib/kubelet/pods/fcb8c646-53fc-462c-8146-476db2e5118d/volumes"
Sep 16 19:20:02.252138 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.252106    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"]
Sep 16 19:20:02.252361 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.252334    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct" podUID="bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f" containerName="payload-pre-processing" containerID="cri-o://e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c" gracePeriod=30
Sep 16 19:20:02.334000 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.333947    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:02.500164 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.500112    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:02.506562 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.506541    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:20:02.543760 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.543726    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"]
Sep 16 19:20:02.547133 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.547103    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ca3f9a-22fsm"]
Sep 16 19:20:02.615195 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.615159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjsl8\" (UniqueName: \"kubernetes.io/projected/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-kube-api-access-tjsl8\") pod \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\" (UID: \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\") "
Sep 16 19:20:02.615355 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.615242    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-plugins-config-volume\") pod \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\" (UID: \"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f\") "
Sep 16 19:20:02.615611 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.615580    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f" (UID: "bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:02.617272 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.617249    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-kube-api-access-tjsl8" (OuterVolumeSpecName: "kube-api-access-tjsl8") pod "bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f" (UID: "bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f"). InnerVolumeSpecName "kube-api-access-tjsl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:02.668428 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.668377    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:02.711345 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.711309    2559 generic.go:356] "Generic (PLEG): container finished" podID="bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f" containerID="e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c" exitCode=0
Sep 16 19:20:02.711500 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.711355    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"
Sep 16 19:20:02.711500 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.711392    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct" event={"ID":"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f","Type":"ContainerDied","Data":"e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c"}
Sep 16 19:20:02.711500 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.711448    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct" event={"ID":"bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f","Type":"ContainerDied","Data":"af79fe0aa66aa3816340411c29d3a0442e998bbeb76f835663264f730780308f"}
Sep 16 19:20:02.711500 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.711473    2559 scope.go:117] "RemoveContainer" containerID="e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c"
Sep 16 19:20:02.715775 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.715750    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:20:02.715889 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.715779    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsl8\" (UniqueName: \"kubernetes.io/projected/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f-kube-api-access-tjsl8\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:20:02.720385 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.720367    2559 scope.go:117] "RemoveContainer" containerID="e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c"
Sep 16 19:20:02.720645 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:20:02.720623    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c\": container with ID starting with e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c not found: ID does not exist" containerID="e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c"
Sep 16 19:20:02.720713 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.720654    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c"} err="failed to get container status \"e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c\": rpc error: code = NotFound desc = could not find container \"e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c\": container with ID starting with e6774cf61fea49f06239c50c335b65d85d4afb2eb665bc1aecb451615855e64c not found: ID does not exist"
Sep 16 19:20:02.735755 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.735728    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"]
Sep 16 19:20:02.740688 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.740631    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-ca3f9a-7d568b8b7d-2lzct"]
Sep 16 19:20:02.919047 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:02.918968    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:03.020065 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:03.019951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:03.974527 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:03.974495    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:04.221808 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:04.221775    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f" path="/var/lib/kubelet/pods/bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f/volumes"
Sep 16 19:20:04.222135 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:04.222120    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6dee68d-c45e-4f4b-9dc5-7b234dfb081d" path="/var/lib/kubelet/pods/d6dee68d-c45e-4f4b-9dc5-7b234dfb081d/volumes"
Sep 16 19:20:09.708051 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:09.707994    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:09.811578 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:09.811511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:09.961244 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:09.961095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:10.304281 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.304197    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"]
Sep 16 19:20:10.304496 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.304484    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6dee68d-c45e-4f4b-9dc5-7b234dfb081d" containerName="revoke-keys"
Sep 16 19:20:10.304540 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.304497    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6dee68d-c45e-4f4b-9dc5-7b234dfb081d" containerName="revoke-keys"
Sep 16 19:20:10.304540 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.304511    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f" containerName="payload-pre-processing"
Sep 16 19:20:10.304540 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.304517    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f" containerName="payload-pre-processing"
Sep 16 19:20:10.304636 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.304566    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdfd0e3e-01df-4ca5-bff3-d5a23c2c5d8f" containerName="payload-pre-processing"
Sep 16 19:20:10.304636 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.304574    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6dee68d-c45e-4f4b-9dc5-7b234dfb081d" containerName="revoke-keys"
Sep 16 19:20:10.307408 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.307391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"
Sep 16 19:20:10.310092 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.310071    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 19:20:10.310242 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.310171    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nmr62\""
Sep 16 19:20:10.317007 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.315966    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"]
Sep 16 19:20:10.377354 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.377312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71ac210-f653-4f15-b1de-d67a556d3549-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6\" (UID: \"a71ac210-f653-4f15-b1de-d67a556d3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"
Sep 16 19:20:10.478307 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.478266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71ac210-f653-4f15-b1de-d67a556d3549-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6\" (UID: \"a71ac210-f653-4f15-b1de-d67a556d3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"
Sep 16 19:20:10.478884 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.478863    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71ac210-f653-4f15-b1de-d67a556d3549-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6\" (UID: \"a71ac210-f653-4f15-b1de-d67a556d3549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"
Sep 16 19:20:10.618864 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.618775    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"
Sep 16 19:20:10.893713 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:10.893689    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"]
Sep 16 19:20:10.894897 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:20:10.894874    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71ac210_f653_4f15_b1de_d67a556d3549.slice/crio-e4cd29c72ec0a52a3fab78882b9c5d66646eace17c16c64ad1a8389fec2b83bd WatchSource:0}: Error finding container e4cd29c72ec0a52a3fab78882b9c5d66646eace17c16c64ad1a8389fec2b83bd: Status 404 returned error can't find the container with id e4cd29c72ec0a52a3fab78882b9c5d66646eace17c16c64ad1a8389fec2b83bd
Sep 16 19:20:11.183470 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:11.183381    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:11.752549 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:11.752509    2559 generic.go:356] "Generic (PLEG): container finished" podID="a71ac210-f653-4f15-b1de-d67a556d3549" containerID="8fcf5047df051e37367176b927fc080481e0d22193b8118342ee18814a0b7ea3" exitCode=0
Sep 16 19:20:11.752712 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:11.752593    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6" event={"ID":"a71ac210-f653-4f15-b1de-d67a556d3549","Type":"ContainerDied","Data":"8fcf5047df051e37367176b927fc080481e0d22193b8118342ee18814a0b7ea3"}
Sep 16 19:20:11.752712 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:11.752641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6" event={"ID":"a71ac210-f653-4f15-b1de-d67a556d3549","Type":"ContainerStarted","Data":"e4cd29c72ec0a52a3fab78882b9c5d66646eace17c16c64ad1a8389fec2b83bd"}
Sep 16 19:20:12.144358 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:12.144325    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:12.872702 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:12.872679    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"
Sep 16 19:20:13.000245 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:13.000206    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71ac210-f653-4f15-b1de-d67a556d3549-maas-api-service-ca\") pod \"a71ac210-f653-4f15-b1de-d67a556d3549\" (UID: \"a71ac210-f653-4f15-b1de-d67a556d3549\") "
Sep 16 19:20:13.000573 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:13.000549    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71ac210-f653-4f15-b1de-d67a556d3549-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a71ac210-f653-4f15-b1de-d67a556d3549" (UID: "a71ac210-f653-4f15-b1de-d67a556d3549"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:13.101136 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:13.101045    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a71ac210-f653-4f15-b1de-d67a556d3549-maas-api-service-ca\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:20:13.762916 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:13.762884    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"
Sep 16 19:20:13.762916 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:13.762901    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6" event={"ID":"a71ac210-f653-4f15-b1de-d67a556d3549","Type":"ContainerDied","Data":"e4cd29c72ec0a52a3fab78882b9c5d66646eace17c16c64ad1a8389fec2b83bd"}
Sep 16 19:20:13.763371 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:13.762947    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4cd29c72ec0a52a3fab78882b9c5d66646eace17c16c64ad1a8389fec2b83bd"
Sep 16 19:20:21.000441 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.000368    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:21.058219 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.058183    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"]
Sep 16 19:20:21.066963 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.066930    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2dc9fc-29826435-r9xd7"]
Sep 16 19:20:21.146271 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.146235    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"]
Sep 16 19:20:21.146483 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.146460    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf" podUID="7ba21cb9-148a-432d-b4c8-c2a8bf7de574" containerName="payload-pre-processing" containerID="cri-o://b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4" gracePeriod=30
Sep 16 19:20:21.190367 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.190327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:21.280405 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.280373    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:21.384639 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.384616    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:20:21.410477 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.410445    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:21.467757 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.467720    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-plugins-config-volume\") pod \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\" (UID: \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\") "
Sep 16 19:20:21.467757 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.467758    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkxxz\" (UniqueName: \"kubernetes.io/projected/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-kube-api-access-jkxxz\") pod \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\" (UID: \"7ba21cb9-148a-432d-b4c8-c2a8bf7de574\") "
Sep 16 19:20:21.468157 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.468133    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ba21cb9-148a-432d-b4c8-c2a8bf7de574" (UID: "7ba21cb9-148a-432d-b4c8-c2a8bf7de574"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:20:21.469938 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.469914    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-kube-api-access-jkxxz" (OuterVolumeSpecName: "kube-api-access-jkxxz") pod "7ba21cb9-148a-432d-b4c8-c2a8bf7de574" (UID: "7ba21cb9-148a-432d-b4c8-c2a8bf7de574"). InnerVolumeSpecName "kube-api-access-jkxxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:21.473387 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.473365    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"]
Sep 16 19:20:21.479369 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.479350    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-2dc9fc-n9tk6"]
Sep 16 19:20:21.516756 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.516660    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:21.568687 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.568659    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-plugins-config-volume\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:20:21.568687 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.568683    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxxz\" (UniqueName: \"kubernetes.io/projected/7ba21cb9-148a-432d-b4c8-c2a8bf7de574-kube-api-access-jkxxz\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:20:21.804636 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.804548    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ba21cb9-148a-432d-b4c8-c2a8bf7de574" containerID="b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4" exitCode=0
Sep 16 19:20:21.804636 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.804596    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"
Sep 16 19:20:21.804837 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.804634    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf" event={"ID":"7ba21cb9-148a-432d-b4c8-c2a8bf7de574","Type":"ContainerDied","Data":"b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4"}
Sep 16 19:20:21.804837 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.804690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf" event={"ID":"7ba21cb9-148a-432d-b4c8-c2a8bf7de574","Type":"ContainerDied","Data":"c5855fd3adda3561dfdeb6944911bbc220520e89d6f599bd4fd06d69a24c922c"}
Sep 16 19:20:21.804837 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.804712    2559 scope.go:117] "RemoveContainer" containerID="b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4"
Sep 16 19:20:21.814230 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.814213    2559 scope.go:117] "RemoveContainer" containerID="b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4"
Sep 16 19:20:21.814496 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:20:21.814475    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4\": container with ID starting with b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4 not found: ID does not exist" containerID="b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4"
Sep 16 19:20:21.814546 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.814504    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4"} err="failed to get container status \"b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4\": rpc error: code = NotFound desc = could not find container \"b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4\": container with ID starting with b0327e3880f515f4350e6d1356a3a433663f3105602ba936a0178ec4f6735fe4 not found: ID does not exist"
Sep 16 19:20:21.830183 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.830159    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"]
Sep 16 19:20:21.834317 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.834298    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2dc9fc-5b5d948bb6-b65lf"]
Sep 16 19:20:21.854460 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:21.854437    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:22.007825 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:22.007793    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:22.222094 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:22.222061    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba21cb9-148a-432d-b4c8-c2a8bf7de574" path="/var/lib/kubelet/pods/7ba21cb9-148a-432d-b4c8-c2a8bf7de574/volumes"
Sep 16 19:20:22.222409 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:22.222395    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a71ac210-f653-4f15-b1de-d67a556d3549" path="/var/lib/kubelet/pods/a71ac210-f653-4f15-b1de-d67a556d3549/volumes"
Sep 16 19:20:22.222651 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:22.222638    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d9787d-a19a-426c-87c5-cf20f192e9b2" path="/var/lib/kubelet/pods/d2d9787d-a19a-426c-87c5-cf20f192e9b2/volumes"
Sep 16 19:20:29.379072 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:29.379039    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:29.472288 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:29.472255    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:29.709990 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:20:29.709942    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:28.515378 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:21:28.515343    2559 scope.go:117] "RemoveContainer" containerID="702388d8d477757f1cdb3884cc4f6184e9c96232d575df396ae89faa38fb49ae"
Sep 16 19:21:28.523218 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:21:28.523196    2559 scope.go:117] "RemoveContainer" containerID="212fc8bda5e8f192faa577ff71a368f4b4f7e1ab273744b7345ba56cb5566eef"
Sep 16 19:21:55.879021 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:21:55.878991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:57.407128 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:21:57.407099    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:56.723834 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:22:56.723756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:56.821558 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:22:56.821522    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:20.366183 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:23:20.366150    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:27.561204 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:27.561172    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:28.177807 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:28.177780    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:24:28.183595 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:28.183574    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9kh6c_bfd9b33d-0abc-495a-b09d-0b2a2f8d5bbe/ovn-acl-logging/0.log"
Sep 16 19:24:28.561605 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:28.561579    2559 scope.go:117] "RemoveContainer" containerID="a62332ea90c9801f91139790fd605b3724b899efd03e91599df278d348be4265"
Sep 16 19:24:55.763073 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.763040    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"]
Sep 16 19:24:55.763659 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.763305    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba21cb9-148a-432d-b4c8-c2a8bf7de574" containerName="payload-pre-processing"
Sep 16 19:24:55.763659 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.763317    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba21cb9-148a-432d-b4c8-c2a8bf7de574" containerName="payload-pre-processing"
Sep 16 19:24:55.763659 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.763340    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71ac210-f653-4f15-b1de-d67a556d3549" containerName="revoke-keys"
Sep 16 19:24:55.763659 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.763345    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71ac210-f653-4f15-b1de-d67a556d3549" containerName="revoke-keys"
Sep 16 19:24:55.763659 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.763396    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba21cb9-148a-432d-b4c8-c2a8bf7de574" containerName="payload-pre-processing"
Sep 16 19:24:55.763659 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.763404    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71ac210-f653-4f15-b1de-d67a556d3549" containerName="revoke-keys"
Sep 16 19:24:55.766236 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.766218    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.769046 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.769020    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-651efd89-kserve-self-signed-certs\""
Sep 16 19:24:55.777418 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.777397    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"]
Sep 16 19:24:55.801609 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.801585    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:55.861925 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.861888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-dshm\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.862115 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.861934    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44v8\" (UniqueName: \"kubernetes.io/projected/5674ca20-d7eb-44d0-8456-6c18ee820e20-kube-api-access-p44v8\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.862115 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.861953    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-tmp-dir\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.862115 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.861992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-model-cache\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.862115 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.862100    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5674ca20-d7eb-44d0-8456-6c18ee820e20-tls-certs\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.862305 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.862145    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-home\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.962843 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.962808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44v8\" (UniqueName: \"kubernetes.io/projected/5674ca20-d7eb-44d0-8456-6c18ee820e20-kube-api-access-p44v8\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.962843 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.962846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-tmp-dir\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.963132 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.962874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-model-cache\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.963132 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.962928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5674ca20-d7eb-44d0-8456-6c18ee820e20-tls-certs\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.963132 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.962955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-home\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.963132 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.963041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-dshm\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.963342 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.963293    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-tmp-dir\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.963405 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.963347    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-model-cache\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.963405 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.963367    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-home\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.965257 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.965232    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-dshm\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.965499 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.965479    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5674ca20-d7eb-44d0-8456-6c18ee820e20-tls-certs\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:55.971440 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:55.971421    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44v8\" (UniqueName: \"kubernetes.io/projected/5674ca20-d7eb-44d0-8456-6c18ee820e20-kube-api-access-p44v8\") pod \"e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") " pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:56.075784 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:56.075700    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:24:56.347417 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:56.347338    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"]
Sep 16 19:24:56.350042 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:24:56.350014    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5674ca20_d7eb_44d0_8456_6c18ee820e20.slice/crio-f490945796245952f15ce0a6f32fb28a5ba0a59f245c52d50c6b1c14cb335c9b WatchSource:0}: Error finding container f490945796245952f15ce0a6f32fb28a5ba0a59f245c52d50c6b1c14cb335c9b: Status 404 returned error can't find the container with id f490945796245952f15ce0a6f32fb28a5ba0a59f245c52d50c6b1c14cb335c9b
Sep 16 19:24:56.962264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:56.962208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c" event={"ID":"5674ca20-d7eb-44d0-8456-6c18ee820e20","Type":"ContainerStarted","Data":"240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52"}
Sep 16 19:24:56.962264 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:56.962267    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c" event={"ID":"5674ca20-d7eb-44d0-8456-6c18ee820e20","Type":"ContainerStarted","Data":"f490945796245952f15ce0a6f32fb28a5ba0a59f245c52d50c6b1c14cb335c9b"}
Sep 16 19:24:56.986831 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:24:56.986780    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c" podStartSLOduration=1.986764853 podStartE2EDuration="1.986764853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:56.985206781 +0000 UTC m=+2729.353262698" watchObservedRunningTime="2026-09-16 19:24:56.986764853 +0000 UTC m=+2729.354820753"
Sep 16 19:25:04.199519 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:04.199409    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:06.076019 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:06.075959    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:25:06.076408 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:06.076038    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:25:06.088744 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:06.088718    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:25:07.018301 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:07.018266    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:25:31.916355 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:31.916318    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"]
Sep 16 19:25:31.961248 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:31.961211    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:33.014586 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:33.014549    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:46.934983 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:46.934919    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c" podUID="5674ca20-d7eb-44d0-8456-6c18ee820e20" containerName="main" containerID="cri-o://240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52" gracePeriod=45
Sep 16 19:25:47.006923 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.006871    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c" podUID="5674ca20-d7eb-44d0-8456-6c18ee820e20" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:25:47.178013 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.177985    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:25:47.184409 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.184384    2559 generic.go:356] "Generic (PLEG): container finished" podID="5674ca20-d7eb-44d0-8456-6c18ee820e20" containerID="240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52" exitCode=0
Sep 16 19:25:47.184549 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.184424    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"
Sep 16 19:25:47.184549 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.184438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c" event={"ID":"5674ca20-d7eb-44d0-8456-6c18ee820e20","Type":"ContainerDied","Data":"240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52"}
Sep 16 19:25:47.184549 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.184475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c" event={"ID":"5674ca20-d7eb-44d0-8456-6c18ee820e20","Type":"ContainerDied","Data":"f490945796245952f15ce0a6f32fb28a5ba0a59f245c52d50c6b1c14cb335c9b"}
Sep 16 19:25:47.184549 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.184500    2559 scope.go:117] "RemoveContainer" containerID="240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52"
Sep 16 19:25:47.192290 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.192267    2559 scope.go:117] "RemoveContainer" containerID="240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52"
Sep 16 19:25:47.192542 ip-10-0-141-134 kubenswrapper[2559]: E0916 19:25:47.192525    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52\": container with ID starting with 240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52 not found: ID does not exist" containerID="240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52"
Sep 16 19:25:47.192594 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.192551    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52"} err="failed to get container status \"240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52\": rpc error: code = NotFound desc = could not find container \"240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52\": container with ID starting with 240e8660ba2924ce21698ec6c7b62319315b4cd8fe9ad537b6e4ba43a7e08c52 not found: ID does not exist"
Sep 16 19:25:47.281255 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281219    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p44v8\" (UniqueName: \"kubernetes.io/projected/5674ca20-d7eb-44d0-8456-6c18ee820e20-kube-api-access-p44v8\") pod \"5674ca20-d7eb-44d0-8456-6c18ee820e20\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") "
Sep 16 19:25:47.281427 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281281    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-home\") pod \"5674ca20-d7eb-44d0-8456-6c18ee820e20\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") "
Sep 16 19:25:47.281427 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281330    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-tmp-dir\") pod \"5674ca20-d7eb-44d0-8456-6c18ee820e20\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") "
Sep 16 19:25:47.281427 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281387    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-dshm\") pod \"5674ca20-d7eb-44d0-8456-6c18ee820e20\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") "
Sep 16 19:25:47.281589 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281433    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-model-cache\") pod \"5674ca20-d7eb-44d0-8456-6c18ee820e20\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") "
Sep 16 19:25:47.281589 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281459    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5674ca20-d7eb-44d0-8456-6c18ee820e20-tls-certs\") pod \"5674ca20-d7eb-44d0-8456-6c18ee820e20\" (UID: \"5674ca20-d7eb-44d0-8456-6c18ee820e20\") "
Sep 16 19:25:47.281589 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281547    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5674ca20-d7eb-44d0-8456-6c18ee820e20" (UID: "5674ca20-d7eb-44d0-8456-6c18ee820e20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:47.281750 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281558    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-home" (OuterVolumeSpecName: "home") pod "5674ca20-d7eb-44d0-8456-6c18ee820e20" (UID: "5674ca20-d7eb-44d0-8456-6c18ee820e20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:47.281750 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281685    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-tmp-dir\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:25:47.281750 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.281713    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-model-cache" (OuterVolumeSpecName: "model-cache") pod "5674ca20-d7eb-44d0-8456-6c18ee820e20" (UID: "5674ca20-d7eb-44d0-8456-6c18ee820e20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:47.283629 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.283599    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5674ca20-d7eb-44d0-8456-6c18ee820e20-kube-api-access-p44v8" (OuterVolumeSpecName: "kube-api-access-p44v8") pod "5674ca20-d7eb-44d0-8456-6c18ee820e20" (UID: "5674ca20-d7eb-44d0-8456-6c18ee820e20"). InnerVolumeSpecName "kube-api-access-p44v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:25:47.283746 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.283649    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-dshm" (OuterVolumeSpecName: "dshm") pod "5674ca20-d7eb-44d0-8456-6c18ee820e20" (UID: "5674ca20-d7eb-44d0-8456-6c18ee820e20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:25:47.283746 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.283653    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5674ca20-d7eb-44d0-8456-6c18ee820e20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5674ca20-d7eb-44d0-8456-6c18ee820e20" (UID: "5674ca20-d7eb-44d0-8456-6c18ee820e20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:25:47.382279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.382234    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-dshm\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:25:47.382279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.382270    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-model-cache\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:25:47.382279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.382283    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5674ca20-d7eb-44d0-8456-6c18ee820e20-tls-certs\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:25:47.382279 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.382291    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44v8\" (UniqueName: \"kubernetes.io/projected/5674ca20-d7eb-44d0-8456-6c18ee820e20-kube-api-access-p44v8\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:25:47.382603 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.382301    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5674ca20-d7eb-44d0-8456-6c18ee820e20-home\") on node \"ip-10-0-141-134.ec2.internal\" DevicePath \"\""
Sep 16 19:25:47.506227 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.506196    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"]
Sep 16 19:25:47.508865 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:47.508840    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-651efd89-kserve-675fbbd4d7-jbj9c"]
Sep 16 19:25:48.221685 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:25:48.221651    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5674ca20-d7eb-44d0-8456-6c18ee820e20" path="/var/lib/kubelet/pods/5674ca20-d7eb-44d0-8456-6c18ee820e20/volumes"
Sep 16 19:26:20.965339 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:20.965305    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66c9cbff69-zkz6l_840523ce-7791-4c2e-8601-99006dd1df9a/authorino/0.log"
Sep 16 19:26:26.646527 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:26.646499    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7kv5l_049a0535-4721-45af-9da0-04e3ff380856/server/0.log"
Sep 16 19:26:26.764063 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:26.764032    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-p8mvw_ca15ef45-43c9-46f2-916e-2f20a3b61184/manager/0.log"
Sep 16 19:26:27.009450 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:27.009420    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85745479c7-qpspw_dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff/copy-manifests/0.log"
Sep 16 19:26:27.017240 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:27.017211    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85745479c7-qpspw_dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff/manager/0.log"
Sep 16 19:26:28.584814 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:28.584781    2559 scope.go:117] "RemoveContainer" containerID="bd7270cd24de01cf0bedfcd76d64baebaf6b21a88f21ff1ebcb89052bd83cfb2"
Sep 16 19:26:28.592652 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:28.592630    2559 scope.go:117] "RemoveContainer" containerID="8fcf5047df051e37367176b927fc080481e0d22193b8118342ee18814a0b7ea3"
Sep 16 19:26:28.599963 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:28.599946    2559 scope.go:117] "RemoveContainer" containerID="fe1ddec1e20e4f8e500372e4880ca633478068a765909f087432a1da5418e44b"
Sep 16 19:26:28.670677 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:28.670648    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66c9cbff69-zkz6l_840523ce-7791-4c2e-8601-99006dd1df9a/authorino/0.log"
Sep 16 19:26:28.986683 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:28.986658    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-hqpnf_d3417344-9071-486d-a528-de677568078e/kuadrant-console-plugin/0.log"
Sep 16 19:26:30.005338 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:30.005308    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d78bff8dd-sp4kx_274a44e2-960f-4baf-9519-d0106a12856f/kube-auth-proxy/0.log"
Sep 16 19:26:30.335858 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:30.335773    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w66f9_3f1972d0-318d-4e9a-b367-6745577a2e9a/payload-processing/0.log"
Sep 16 19:26:31.088851 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:31.088818    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-4vpp2_bf79e50e-a09a-4b24-8db6-900f209eb414/main/0.log"
Sep 16 19:26:31.197262 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:31.197231    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f759ff_cb802bbf-8a7e-47fb-83e7-3416b556f4c2/main/0.log"
Sep 16 19:26:31.411494 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:31.411403    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-czfjm_cb38dfd2-ec66-4fa9-b025-bb4cc22f967d/main/0.log"
Sep 16 19:26:37.021916 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.021879    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mrp4v/must-gather-nrwft"]
Sep 16 19:26:37.022318 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.022175    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5674ca20-d7eb-44d0-8456-6c18ee820e20" containerName="main"
Sep 16 19:26:37.022318 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.022186    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5674ca20-d7eb-44d0-8456-6c18ee820e20" containerName="main"
Sep 16 19:26:37.022318 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.022237    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5674ca20-d7eb-44d0-8456-6c18ee820e20" containerName="main"
Sep 16 19:26:37.026759 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.026739    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mrp4v/must-gather-nrwft"
Sep 16 19:26:37.031006 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.030984    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mrp4v\"/\"default-dockercfg-nn9pg\""
Sep 16 19:26:37.031146 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.031131    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mrp4v\"/\"openshift-service-ca.crt\""
Sep 16 19:26:37.031252 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.031234    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mrp4v\"/\"kube-root-ca.crt\""
Sep 16 19:26:37.040265 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.040243    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mrp4v/must-gather-nrwft"]
Sep 16 19:26:37.041843 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.041821    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:37.068234 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.068211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f506906b-6f1f-48d0-be97-0047273a7fa7-must-gather-output\") pod \"must-gather-nrwft\" (UID: \"f506906b-6f1f-48d0-be97-0047273a7fa7\") " pod="openshift-must-gather-mrp4v/must-gather-nrwft"
Sep 16 19:26:37.068328 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.068251    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t6rl\" (UniqueName: \"kubernetes.io/projected/f506906b-6f1f-48d0-be97-0047273a7fa7-kube-api-access-2t6rl\") pod \"must-gather-nrwft\" (UID: \"f506906b-6f1f-48d0-be97-0047273a7fa7\") " pod="openshift-must-gather-mrp4v/must-gather-nrwft"
Sep 16 19:26:37.141299 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.141269    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:37.169020 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.168989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f506906b-6f1f-48d0-be97-0047273a7fa7-must-gather-output\") pod \"must-gather-nrwft\" (UID: \"f506906b-6f1f-48d0-be97-0047273a7fa7\") " pod="openshift-must-gather-mrp4v/must-gather-nrwft"
Sep 16 19:26:37.169140 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.169036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t6rl\" (UniqueName: \"kubernetes.io/projected/f506906b-6f1f-48d0-be97-0047273a7fa7-kube-api-access-2t6rl\") pod \"must-gather-nrwft\" (UID: \"f506906b-6f1f-48d0-be97-0047273a7fa7\") " pod="openshift-must-gather-mrp4v/must-gather-nrwft"
Sep 16 19:26:37.169325 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.169301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f506906b-6f1f-48d0-be97-0047273a7fa7-must-gather-output\") pod \"must-gather-nrwft\" (UID: \"f506906b-6f1f-48d0-be97-0047273a7fa7\") " pod="openshift-must-gather-mrp4v/must-gather-nrwft"
Sep 16 19:26:37.179215 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.179195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t6rl\" (UniqueName: \"kubernetes.io/projected/f506906b-6f1f-48d0-be97-0047273a7fa7-kube-api-access-2t6rl\") pod \"must-gather-nrwft\" (UID: \"f506906b-6f1f-48d0-be97-0047273a7fa7\") " pod="openshift-must-gather-mrp4v/must-gather-nrwft"
Sep 16 19:26:37.335991 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.335893    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mrp4v/must-gather-nrwft"
Sep 16 19:26:37.606595 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.606535    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mrp4v/must-gather-nrwft"]
Sep 16 19:26:37.608670 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:26:37.608644    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf506906b_6f1f_48d0_be97_0047273a7fa7.slice/crio-e2b9f28ed4f267ce2c0ed9f9eabbc162440137d80d8012c177f5ba78e98933da WatchSource:0}: Error finding container e2b9f28ed4f267ce2c0ed9f9eabbc162440137d80d8012c177f5ba78e98933da: Status 404 returned error can't find the container with id e2b9f28ed4f267ce2c0ed9f9eabbc162440137d80d8012c177f5ba78e98933da
Sep 16 19:26:37.610610 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:37.610590    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:26:38.407097 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:38.407057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mrp4v/must-gather-nrwft" event={"ID":"f506906b-6f1f-48d0-be97-0047273a7fa7","Type":"ContainerStarted","Data":"e2b9f28ed4f267ce2c0ed9f9eabbc162440137d80d8012c177f5ba78e98933da"}
Sep 16 19:26:39.416661 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:39.416620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mrp4v/must-gather-nrwft" event={"ID":"f506906b-6f1f-48d0-be97-0047273a7fa7","Type":"ContainerStarted","Data":"83cdf7405f9ad2e669285683082cb2d44a444ed59badeae6fa6ad0a60bc91910"}
Sep 16 19:26:39.417231 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:39.417203    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mrp4v/must-gather-nrwft" event={"ID":"f506906b-6f1f-48d0-be97-0047273a7fa7","Type":"ContainerStarted","Data":"b4e32166450efc47c794a32aec3c4c4f21e870857e0c9eb1ae0233d974575d67"}
Sep 16 19:26:39.440617 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:39.440562    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mrp4v/must-gather-nrwft" podStartSLOduration=1.597535911 podStartE2EDuration="2.440545536s" totalImagesPullingTime="843.009625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:39.439711429 +0000 UTC m=+2831.807767330" watchObservedRunningTime="2026-09-16 19:26:39.440545536 +0000 UTC m=+2831.808601460"
Sep 16 19:26:40.228852 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:40.228824    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mlhxh_9162f61b-6af4-40e3-b4c5-1c7704c48322/global-pull-secret-syncer/0.log"
Sep 16 19:26:40.286336 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:40.286303    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log"
Sep 16 19:26:40.435722 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:40.435690    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-134.ec2.internal_987b3dcc62388b87e3077d94e8941aa6/haproxy/0.log"
Sep 16 19:26:41.173330 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:41.173296    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mqdvl_ef01009f-d147-475f-9115-275e01b03529/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:44.263491 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:44.261827    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-66c9cbff69-zkz6l_840523ce-7791-4c2e-8601-99006dd1df9a/authorino/0.log"
Sep 16 19:26:44.358775 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:44.358731    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-hqpnf_d3417344-9071-486d-a528-de677568078e/kuadrant-console-plugin/0.log"
Sep 16 19:26:46.704913 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:46.704872    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xg6dr_e5541b1b-46c2-4be3-8fc0-3da9cc0e0002/node-exporter/0.log"
Sep 16 19:26:46.727167 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:46.727122    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xg6dr_e5541b1b-46c2-4be3-8fc0-3da9cc0e0002/kube-rbac-proxy/0.log"
Sep 16 19:26:46.750854 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:46.750815    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xg6dr_e5541b1b-46c2-4be3-8fc0-3da9cc0e0002/init-textfile/0.log"
Sep 16 19:26:51.042382 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.042336    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"]
Sep 16 19:26:51.048526 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.048499    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.055340 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.055309    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"]
Sep 16 19:26:51.111955 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.111904    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-podres\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.112168 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.112043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dpb6\" (UniqueName: \"kubernetes.io/projected/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-kube-api-access-6dpb6\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.112168 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.112079    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-sys\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.112306 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.112153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-lib-modules\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.112306 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.112217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-proc\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.213568    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-podres\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.213652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dpb6\" (UniqueName: \"kubernetes.io/projected/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-kube-api-access-6dpb6\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.213683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-sys\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.213716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-lib-modules\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.213746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-proc\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.213875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-proc\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.214004    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-podres\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.214360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-sys\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.214492 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.214463    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-lib-modules\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.227386 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.227352    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dpb6\" (UniqueName: \"kubernetes.io/projected/f6d4a257-751b-4550-8ef2-7f84bfcf9de8-kube-api-access-6dpb6\") pod \"perf-node-gather-daemonset-tzn2m\" (UID: \"f6d4a257-751b-4550-8ef2-7f84bfcf9de8\") " pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.363770 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.363692    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:51.646698 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:51.646601    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"]
Sep 16 19:26:51.653450 ip-10-0-141-134 kubenswrapper[2559]: W0916 19:26:51.653406    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf6d4a257_751b_4550_8ef2_7f84bfcf9de8.slice/crio-a75ced6442b63ffc2ab2da49f0c816e7cd29682a12d577a8624e0f3164e0ee33 WatchSource:0}: Error finding container a75ced6442b63ffc2ab2da49f0c816e7cd29682a12d577a8624e0f3164e0ee33: Status 404 returned error can't find the container with id a75ced6442b63ffc2ab2da49f0c816e7cd29682a12d577a8624e0f3164e0ee33
Sep 16 19:26:52.308430 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:52.308399    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4tzv7_3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93/dns/0.log"
Sep 16 19:26:52.330902 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:52.330845    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4tzv7_3cc3f93a-7244-4ef2-80c6-ba4af3ce9d93/kube-rbac-proxy/0.log"
Sep 16 19:26:52.499017 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:52.498965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m" event={"ID":"f6d4a257-751b-4550-8ef2-7f84bfcf9de8","Type":"ContainerStarted","Data":"5dfc1d2b8b8c16237839dce1a3eb6f3d64f16d50361016189ce165628d6fe81e"}
Sep 16 19:26:52.499218 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:52.499036    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m" event={"ID":"f6d4a257-751b-4550-8ef2-7f84bfcf9de8","Type":"ContainerStarted","Data":"a75ced6442b63ffc2ab2da49f0c816e7cd29682a12d577a8624e0f3164e0ee33"}
Sep 16 19:26:52.499218 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:52.499188    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:52.509016 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:52.508992    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qk4dv_fe2346ee-9571-4d40-bc5d-cb7292a5efd3/dns-node-resolver/0.log"
Sep 16 19:26:52.520417 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:52.520373    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m" podStartSLOduration=1.520356401 podStartE2EDuration="1.520356401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:52.519916316 +0000 UTC m=+2844.887972239" watchObservedRunningTime="2026-09-16 19:26:52.520356401 +0000 UTC m=+2844.888412301"
Sep 16 19:26:53.224015 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:53.223984    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8xtq5_ac745a81-e369-4633-b72d-a2e3a6ca030d/node-ca/0.log"
Sep 16 19:26:54.758519 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:54.758478    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d78bff8dd-sp4kx_274a44e2-960f-4baf-9519-d0106a12856f/kube-auth-proxy/0.log"
Sep 16 19:26:54.868647 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:54.868615    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w66f9_3f1972d0-318d-4e9a-b367-6745577a2e9a/payload-processing/0.log"
Sep 16 19:26:55.863795 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:55.863764    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xqbwk_d3342a0b-c9c9-46bc-beab-8fd7574ef11b/serve-healthcheck-canary/0.log"
Sep 16 19:26:56.436501 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:56.436473    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67ssr_a577c313-7fc7-4356-96bc-e9efeb8c0591/kube-rbac-proxy/0.log"
Sep 16 19:26:56.469232 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:56.469205    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67ssr_a577c313-7fc7-4356-96bc-e9efeb8c0591/exporter/0.log"
Sep 16 19:26:56.512508 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:56.512481    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67ssr_a577c313-7fc7-4356-96bc-e9efeb8c0591/extractor/0.log"
Sep 16 19:26:58.516585 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:58.516559    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mrp4v/perf-node-gather-daemonset-tzn2m"
Sep 16 19:26:59.227896 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:59.227857    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7kv5l_049a0535-4721-45af-9da0-04e3ff380856/server/0.log"
Sep 16 19:26:59.340262 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:59.340228    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-p8mvw_ca15ef45-43c9-46f2-916e-2f20a3b61184/manager/0.log"
Sep 16 19:26:59.417473 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:59.417442    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85745479c7-qpspw_dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff/manager/0.log"
Sep 16 19:26:59.437875 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:26:59.437841    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85745479c7-qpspw_dd3ca9f7-c181-4ef3-abb4-6cdf27ee7fff/copy-manifests/0.log"
Sep 16 19:27:00.985254 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:00.985212    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f86d45999-zx52x_1d35ced6-b776-4ff3-981c-fc3679d0fca2/manager/0.log"
Sep 16 19:27:01.112879 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:01.112852    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d78bff8dd-sp4kx_274a44e2-960f-4baf-9519-d0106a12856f/kube-auth-proxy/0.log"
Sep 16 19:27:01.246598 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:01.246525    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w66f9_3f1972d0-318d-4e9a-b367-6745577a2e9a/payload-processing/0.log"
Sep 16 19:27:03.198505 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:03.198435    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d78bff8dd-sp4kx_274a44e2-960f-4baf-9519-d0106a12856f/kube-auth-proxy/0.log"
Sep 16 19:27:03.274322 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:03.274292    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-w66f9_3f1972d0-318d-4e9a-b367-6745577a2e9a/payload-processing/0.log"
Sep 16 19:27:08.044444 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:08.044414    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5qcp8_ffb6e126-98d0-45f8-869b-1e2563e38468/kube-multus/0.log"
Sep 16 19:27:08.285604 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:08.285567    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2smj_74a3ac79-2f44-4249-8c19-77427e9073a6/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:27:08.316674 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:08.316607    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2smj_74a3ac79-2f44-4249-8c19-77427e9073a6/egress-router-binary-copy/0.log"
Sep 16 19:27:08.347817 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:08.347785    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2smj_74a3ac79-2f44-4249-8c19-77427e9073a6/cni-plugins/0.log"
Sep 16 19:27:08.376535 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:08.376509    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2smj_74a3ac79-2f44-4249-8c19-77427e9073a6/bond-cni-plugin/0.log"
Sep 16 19:27:08.408380 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:08.408347    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2smj_74a3ac79-2f44-4249-8c19-77427e9073a6/routeoverride-cni/0.log"
Sep 16 19:27:08.433340 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:08.433315    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2smj_74a3ac79-2f44-4249-8c19-77427e9073a6/whereabouts-cni-bincopy/0.log"
Sep 16 19:27:08.456444 ip-10-0-141-134 kubenswrapper[2559]: I0916 19:27:08.456413    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2smj_74a3ac79-2f44-4249-8c19-77427e9073a6/whereabouts-cni/0.log"